The Crowdstrike software update

[cc333]

Computers allow health care providers to share information across the departments of large hospitals and other associated hospitals as well as among both "on campus" and other location physicians and health care providers. That means extensive test results, historical treatment and more.

In a manual system, the data has to go by mail or by sneakernet to get to those who need it. In the meantime, the patient can die.

This is true, of course!

I think a balanced approach is good. Have all the newfangled computer systems as they do, but keep a paper backup for emergencies where their shiny computer gizmos go down, as they did last week.

For better or worse, the world has become so dependent on computers and the Internet, it's very difficult to go without, but I think it's important to continue the "old-fashioned" systems in parallel for times exactly like these.

As for all that AM Stereo stuff, how did that come up? I have a feeling it was my fault....😳

c

 

[Kelly A]

IMHO AM stereo could have extended some full service ACs, classic country stations, and possibly oldies by a decade or more. According to amstereo.org, Kahn* had a system that worked "well" in the 1950's. Could the FCC not have adopted it because it was promoting FM? I am surprised some conspiracy folks haven't had a field day with this.

Besides what David already said; tuner manufacturers weren't interested in demodulating the Kahn ISB system in fear of being added to the long list of his other lawsuits. Sony had a half ass receiver that would come close to demodulating ISB, but not within the specs Leonard claimed. Sony and Carver tried to demodulate ISB, but as soon as word got out, Leonard would send them a stop letter. The only receiver Kahn approved, was a modified Radio Shack plastic case component tuner with a homebrew perfboard scabbed onto the receiver that Leonard hand made. The irony was that Tandy sent Leonard a warning that he wasn't authorized to modify any equipment Tandy/RS sold.

*If had a "workable" AM stereo in the 1950s and couldn't get it accepted, then 30 years later the commission picked another system I would be Mad too.

The difference at the time was you had the big company owners of AM stations who didn't want anyone to mess with their formula. 'We're number one so there's no need trying to change. FM, on the other had was not even on most consumers shopping list, so stereo for the most part, came with the band. Plus, the FM stereo method was from RCA. They had very impressive engineers back in the day, including one who also worked on the FM stereo project. Leonard Kahn.

Another "mad scientist" was Tesla, I saw a History Channel" show about him, and apparently he had sued Marconi over patent infringement for radio. His suit won after he died. If Tesla, Marconi or someone else could have figured out a way around AM skip at night and overcome few physics laws that would have been a patent worth a lot.

There are some great books on Tesla's working relationship to guys like George Westinghouse and Thomas Edison, who both screwed him.

 

[Kelly A]

So, I got some additional technical details on what happened with the CrowdStrike debacle. Apparently the bad code was indeed in the Windows Kernal. Once the kernal was polluted, Windows wouldn't boot. CrowdStrike caught the problem pretty quickly and came up with a fix. The problem was, unless the affected machine would boot, the fix wouldn't stick. At one point CrowdStrike recommended to Microsoft to tell their customers to try rebooting at least 22 times on the chance the modified kernal would get through. Ultimately some customers had to use an old fashioned boot disc in order to boot the machine so the fix could be applied. Depending on the number of licenses within an organization, that could be a lot of machines.

 

[Mark Roberts]

A fairly non-technical analysis of the Crowdstrike debacle has been written by one of the giants of modern computer science, Bruce Schneier, who focuses on the neglect of resilience in pursuit of profit and the lack of consequences for same. Schneier and his co-author contrast resilience with "brittleness":

But brittleness is profitable only when everything is working. When a brittle system fails, it fails badly. The cost of failure to a company like CrowdStrike is a fraction of the cost to the global economy. And there will be a next CrowdStrike, and one after that. The market rewards short-term profit-maximizing systems, and doesn’t sufficiently penalize such companies for the impact their mistakes can have. (Stock prices depress only temporarily. Regulatory penalties are minor. Class-action lawsuits settle. Insurance blunts financial losses.) It’s not even clear that the information technology industry could exist in its current form if it had to take into account all the risks such brittleness causes.

A corollary is that you can't count on businesses to do the right thing; regulation and enhanced liability are not the total solution but would be a start.

The CrowdStrike Outage and Market-Driven Brittleness

The outage is another consequence of companies’ sacrifice of resilience for expediency.

www.lawfaremedia.org

 

[boombox4]

So, I got some additional technical details on what happened with the CrowdStrike debacle. Apparently the bad code was indeed in the Windows Kernal. Once the kernal was polluted, Windows wouldn't boot. CrowdStrike caught the problem pretty quickly and came up with a fix. The problem was, unless the affected machine would boot, the fix wouldn't stick. At one point CrowdStrike recommended to Microsoft to tell their customers to try rebooting at least 22 times on the chance the modified kernal would get through. Ultimately some customers had to use an old fashioned boot disc in order to boot the machine so the fix could be applied. Depending on the number of licenses within an organization, that could be a lot of machines.

Boot disks still exist?

 

[boombox4]

A fairly non-technical analysis of the Crowdstrike debacle has been written by one of the giants of modern computer science, Bruce Schneier, who focuses on the neglect of resilience in pursuit of profit and the lack of consequences for same. Schneier and his co-author contrast resilience with "brittleness":



A corollary is that you can't count on businesses to do the right thing; regulation and enhanced liability are not the total solution but would be a start.

The CrowdStrike Outage and Market-Driven Brittleness

The outage is another consequence of companies’ sacrifice of resilience for expediency.

www.lawfaremedia.org

By any chance would he have been referring to the seemingly current software business model, where you turn out unproven or partially proven software (ostensibly to get a leg up on the competition), and update update update update update update update update update update update update update to fix it later?

My Millennial cousins who are really into computer games say that's how that particular industry seems to have gone.

EDIT: After reading the article, yes, that's how it looks. The writer's constantly updated house corollary seems to be appropriate.:
"Imagine a house where the drywall, flooring, fireplace, and light fixtures are all made by companies that need continuous access and whose failures would cause the house to collapse. You’d never set foot in such a structure, yet that’s how software systems are built."

 

[davideduardo]

Boot disks still exist?

Yes, they can be USB drives or USB SSDs in some cases. However, most consumer versions of Windows will not boot from external drives. I believe the enterprise level versions can with certain restrictions.

Consumer versions won't boot that way to prevent a data thief bring a windows install on a flash drive and entering a computer to steal or access data.

I have a card in one slot of my main computer for www.worldradiohistory.com which has 4 x 1tb m.2 SSDs on it. In a four day cycle, they become clones of the boot drive and can be used in case of an emergency, such as an install gone wrong. All I have to do is change the boot drive in BIOS upon booting and I revert to the last good install.

 

[cc333]

"Imagine a house where the drywall, flooring, fireplace, and light fixtures are all made by companies that need continuous access and whose failures would cause the house to collapse. You’d never set foot in such a structure, yet that’s how software systems are built."

That's exactly how the modern computer industry seems to operate. It's sad because it's wiped out most of the fun of using computers for me, since everything's half-broken and constantly updating for no reason.

To that end, most updates rarely, if ever, seem to actually offer any meaningful improvements anymore; instead, they usually destabilize and reduce the performance – sometimes significantly – of a previously reliable software package, often in the name of "security" (I guess a system that's broken and inoperable is pretty "secure", since nobody, including hackers, can access it).

This Crowdstrike update is a perfect, if extreme case in point of the dangers of constantly rushing out potentially broken updates: the update rendered previously stable and reliable computers completely broken.

most consumer versions of Windows will not boot from external drives.

This is generally true, but with some clever hackery, some (particularly older) versions can be made to boot from removable media.

For example, years ago, I made a CD-ROM that booted a live version of Windows 98. After much trial and error, it actually worked amazingly well for what it was, although it was locked to one specific configuration of one specific machine (long since dead), and since the file system was read only, nothing could be installed, removed or configured, including device drivers (I think I even set up a RAM disk for the temp folder so programs that normally wouldn't run on a read-only disk could work). To work around this immutable nature, I'd have to boot Windows normally from a hard drive (the "master" install), install, remove or configure whatever programs I want, and remake the CD-ROM using that master install as the source for the new CD. It wasn't terribly useful with all these limitations and caveats, but it was a fun proof of concept. Not to mention that it was completely immune from viruses and such, because they couldn't install themselves, nor could they permanently modify any system files!

c

 

[boombox4]

That's exactly how the modern computer industry seems to operate. It's sad because it's wiped out most of the fun of using computers for me, since everything's half-broken and constantly updating for no reason.

To that end, most updates rarely, if ever, seem to actually offer any meaningful improvements anymore; instead, they usually destabilize and reduce the performance – sometimes significantly – of a previously reliable software package, often in the name of "security" (I guess a system that's broken and inoperable is pretty "secure", since nobody, including hackers, can access it).

This Crowdstrike update is a perfect, if extreme case in point of the dangers of constantly rushing out potentially broken updates: the update rendered previously stable and reliable computers completely broken.

I share your observations about the increasingly relentless updating, which used to be several times a year, and now takes place sometimes more than once a week. And if you track your hard drive usage, each update is well over a gigabyte. And like you noticed, the apps and programs don't seem to work any better, aside from a few extra features here and there that get added.

That said, it all begs the question: Is this the best that American tech can come up with? Is this really the business model they think is best for the end user?

I feel for the people dealing with the broken computers due to this CrowdStrike issue, really. All that revenue lost, all that time taken to fix the damage. And when something like this affects hospitals, and 9-1-1 systems, that's serious stuff. There's got to be a better way than this.

 

[xmtrland]

What alot of you are missing is that Crowdstrike is protecting against bad actors. Just like other virus software, Malwarebytes and Norton. Virus detection software needs frequent updates to take into account new exploits. it's how Crowdstrike integrates with windows that made this update an issue.

Is Crowdstrike the best America can offer? A bunch of fortune 500 companies, Hospitals and others thought so. I bet they are re evaluating what is out there now.

It is interesting that some took longer than other to recover. Over all a wake up call to re evaluate networked computer protection.

 

[cc333]

What alot of you are missing is that Crowdstrike is protecting against bad actors. Just like other virus software, Malwarebytes and Norton. Virus detection software needs frequent updates to take into account new exploits. it's how Crowdstrike integrates with windows that made this update an issue.

Agreed. Frequent updates for antivirus/anti malware/anti hacker software is important and makes sense. But why do I need to update my word processor or my music player every other day?

c

 

[Kelly A]

What alot of you are missing is that Crowdstrike is protecting against bad actors. Just like other virus software, Malwarebytes and Norton. Virus detection software needs frequent updates to take into account new exploits. it's how Crowdstrike integrates with windows that made this update an issue.

Is Crowdstrike the best America can offer? A bunch of fortune 500 companies, Hospitals and others thought so. I bet they are re evaluating what is out there now.

It is interesting that some took longer than other to recover. Over all a wake up call to re evaluate networked computer protection.

I think of this situation similar to the Jackinthebox E.coli incident. Jackinthebox learned a hard lesson about not paying close enough attention to food safety, and they haven't had anything similar since. Now, if like many on this board seem to lean into the view that CrowdStrike deserves punishment and loss of long term business for not checking their code, that to me seems unreasonable.
Like my JinB example, I'll bet CrowdStrike will be a much better cybersecurity company after this incident is a distant memory.

 

[boombox4]

What alot of you are missing is that Crowdstrike is protecting against bad actors. Just like other virus software, Malwarebytes and Norton. Virus detection software needs frequent updates to take into account new exploits. it's how Crowdstrike integrates with windows that made this update an issue.

Is Crowdstrike the best America can offer? A bunch of fortune 500 companies, Hospitals and others thought so. I bet they are re evaluating what is out there now.

It is interesting that some took longer than other to recover. Over all a wake up call to re evaluate networked computer protection.

Understood, but my own reference upthread, to "endless updates", wasn't in regard to antivirus updates -- which, with my OS, have always been daily, and almost always unobtrusive when loading. I'm talking the other updates, the ones often referred to as "quality" updates -- the kind that can remove menu functions, move them to another place in the app, and then later on, they move them back to where they were to begin with. That kind of thing.

It's overkill for many of us. A lot of it is stuff that's best left to an app store.

On your overall thought that computer security is valuable -- totally agreed. I had a virus attack in the 2000's and it took a month to clear it up. That was my lesson in computer security -- I learned about it the hard way. Luckily, I lost no data. In today's malware climate, people lose more than data. So in the big picture, yes, computer security is of the highest importance.

 

[boombox4]

I think of this situation similar to the Jackinthebox E.coli incident. Jackinthebox learned a hard lesson about not paying close enough attention to food safety, and they haven't had anything similar since. Now, if like many on this board seem to lean into the view that CrowdStrike deserves punishment and loss of long term business for not checking their code, that to me seems unreasonable.
Like my JinB example, I'll bet CrowdStrike will be a much better cybersecurity company after this incident is a distant memory.

I agree. And I doubt any lawsuit will amount to anything. The complex issues surrounding the failures, and the incredibly high costs of litigation would outweigh any 'gain' in monetary damages, if any party succeeded in suing Crowdstrike (Federal Court can be extremely costly). They'll probably improve their process, just like the burger chain did. I read a story that MS is changing some things, too.

 

[davideduardo]

This entire discussion brings up the issue of government detection and prosecution of the Internet thieves who sabotage sites and steal user information. If millions are spent on, to use a recent example, possibly exaggerated property value estimates, why is there not an even greater effort to thwart digital vandals and thieves?

Nearly every one of us has had someone put a charge on our credit card, steal personal data or worse. A huge campaign should be instituted and punishments made the same as car theft, robbing a bank or breaking & entering.

 

[Kelly A]

This entire discussion brings up the issue of government detection and prosecution of the Internet thieves who sabotage sites and steal user information. If millions are spent on, to use a recent example, possibly exaggerated property value estimates, why is there not an even greater effort to thwart digital vandals and thieves?

Nearly every one of us has had someone put a charge on our credit card, steal personal data or worse. A huge campaign should be instituted and punishments made the same as car theft, robbing a bank or breaking & entering.

That's a great point. How many hundreds of millions are lost worldwide every month from ransom and malware? Other than internally, where is the outrage and demands for action when many many more systems are hacked and money stolen than this one unique, public incident?

 

[Larry]

This entire discussion brings up the issue of government detection and prosecution of the Internet thieves who sabotage sites and steal user information. If millions are spent on, to use a recent example, possibly exaggerated property value estimates, why is there not an even greater effort to thwart digital vandals and thieves?

My hunch is they are difficult to detect and prosecute because so many of the “bad actors” are international and not inside the U,S.

 

[boombox4]

Then you have the threat of foreign hackers, hacking into our infrastructure.

They're all computer crimes. I'm sure that malware and ransomware is illegal. But the problem is that a lot of the bad actors are overseas, or expensive to track down There are a lot of other, related crimes that really don't get prosecuted that devastate people's lives. Elderly people can get scammed out of their life savings and even go hundreds of thousands of dollars into debt in the process, and some of those crimes never get resolved.

The point being that although computer crime should be a high priority for law enforcement, there are just so many agents and so much money budgeted for investigating crimes, and sometimes these computer / tech / scam via the phone crimes just never get dealt with.

 

[Kelly A]

Then you have the threat of foreign hackers, hacking into our infrastructure.

They're all computer crimes. I'm sure that malware and ransomware is illegal. But the problem is that a lot of the bad actors are overseas, or expensive to track down There are a lot of other, related crimes that really don't get prosecuted that devastate people's lives. Elderly people can get scammed out of their life savings and even go hundreds of thousands of dollars into debt in the process, and some of those crimes never get resolved.

The point being that although computer crime should be a high priority for law enforcement, there are just so many agents and so much money budgeted for investigating crimes, and sometimes these computer / tech / scam via the phone crimes just never get dealt with.

A few years ago a TV network I was in charge of had a disgruntled factory tech support rep backdoor into our Media Asset Management system and randomly turned off services. They did the same over at Fox.
Certainly not as bad as holding the archive for ransom or deleting critical files including a database, but the malicious work did slow things down and created problems for our engineers. We caught them because one of my guys had the forethought to check the remote access logs from the night prior and saw that their support IP had been in our system the prior night with no request to do so from our side. We even managed to determine whose machine at the manufacturer support desk was used to access our system.
I was told to contact the FBI and file a complaint against the support rep. According to the FBI agent I spoke with, there is a minimum $5,000 damages threshold before the feds get involved and even then, there are so many different domestic and foreign bad guys ripping people off, that they probably won't do anything to a disgruntled support worker.
When I alerted the CEO of the MAM provider, they fired the support agent immediately. But that's the worst happened to them.

 

[leethalweapon]

The report is in…

CrowdStrike reveals root cause of global outage, and it's a mistake first-year programming students learn about

CrowdStrike releases its analysis of last month's global IT outage, revealing an "embarrassing" mistake experts say first-year programming students are taught how to avoid.

www.abc.net.au

When updates are sent, it changes the location or the number of sensors to check for a potential attack.

In this instance, Falcon expected the update to have 20 input fields, but it had 21 input fields.

This "count mismatch" is what caused the global crash, CrowdStrike said.