Crowdstrike has released their preliminary "incident review" on the collapse of several million Windows machines, which their software caused.

As I surmised in my podcast a couple of days after it happened, as soon as the tracebacks were posted publicly and it was clear that the update itself wasn't a signed driver update, the fault had been in the code for quite some time.  We now know from their own disclosure that it dated to at least February 28th.

Timeline of Events: Testing and Rollout of the InterProcessCommunication (IPC) Template Type
Sensor Content Release: On February 28, 2024, sensor 7.11 was made generally available to customers, introducing a new IPC Template Type to detect novel attack techniques that abuse Named Pipes. This release followed all Sensor Content testing procedures outlined above in the Sensor Content section.

Which, as we now know, obviously failed to catch that a malformed template could result in an attempted dereference of a structure that was null and thus invalid, and thus produce a machine check since the context is in the kernel at the time (e.g. BSOD, kernel panic, whatever you care to call it.)

The rest of the document goes on to talk about their alleged "protections" against subsequent similar events, as well as remediation.

The root problem isn't that someone has built software to run in this fashion, although that's a separate point of debate and its worthy of being run to ground because allowing something called a "driver" to be controlled by an external set of data is potentially problematic.  The reason for this to be a point of debate is one of scope-of-work when it comes to this sort of software in the first place, and whether it ought to exist.  That's a debate for a different article, and perhaps one I'll take on at some point.

No, the root problem is architectural in the entities that ran into this.

Simply put it is ridiculously stupid from an IT infrastructure perspective to be in a position where such a tool is "required" anywhere except at the interface between internal resources and the uncontrolled wider Internet.

Contemplate two basic IT models:

• No machine internal can "see" anything beyond the enterprise except that which is approved and necessary for a business operation.  All computers within an entity access those outside services via a gateway which contains whatever security provisions are required (e.g. "firewalls", etc.) on a permitted-only-when-required basis.  If an employee works from home, for example, the device(s) they use are configured to only connect via a VPN which thus effectively places that device "inside" the enterprise network, that machine contains only the approved and allowed software and resources and is owned and managed by the firm.  Windows, as an example, has this capacity through what is called "Group Policy" and it is made clear that not only is compliance monitored but attempted violations, even something as simple as plugging in a USB drive, are instant firing offenses.
  
  
• All machines can "see" anything anywhere on the global internet except for specifically-blacklisted things (e.g. Pornhub.)  There may be specific "internal" higher-security rings of access but interconnections exist between these and the "ordinary" levels.  This computing model relies on each and every device being secure against threats.

The problem with the first, which is basically never practiced today, is that it instantly exposes and prohibits thins such as sitting on Facebook at work.  Why?  Because that's not a permitted destination and function so if you try it the gateway flags the source and suddenly you're facing a write-up or even dismissal, and it doesn't work besides.

But that begs the obvious question: For what purpose is the enterprise's computer network operated?  Is it there and funded for the pleasure and personal abuse of employees or does it exist and is it funded for the explicit purpose of advancing the business?  No, the two are not the same and its a binary decision.

The simple reality of the matter is that firms like Cloudstrike exist because they enable the second model which is defective on its face.  The issue isn't a software bug -- its that there's no "mass market" if the product has one computer on which it runs and has no complex requirements in the first place because the traffic and circumstances under which it would act to interdict a "bad thing" can't arise because the machine(s) inside the enterprise are for the purpose of advancing the business of the company only and circumvention of this mandate brings immediate and summary reprisal including termination which is quite-easy to discover and enforce since it all has to go through one place to get "outside" the perimeter and thus is quite easy and reasonable to monitor and interdict.

Simply put if you design IT infrastructure intelligently there is no reason for a checkout stand at WalMart or Home Depot to have such a thing as Crowdstrike loaded on it because there is no circumstance under which it can "see" outside the facility and thus there are no accessible means by which it can have anything nasty loaded to it or executed upon it.

Witness the "poll books" that were disrupted by this incident during early voting in a couple of jurisdictions.  There is no reasonable design of an IT infrastructure for poll books that requires this.  How did we manage this when the poll books of registered voters were printed on paper?  We did, you know -- so how different is an electronic poll book that has said data loaded before the election and then the device is physically sealed in a tamper-evident way and if you require multiple terminals they are all on a network that has no scope beyond the room in which they are whatsoever by any means.

Thus there is no "threat profile" from external access because no such access exists.

I watched so-called "expert" after "expert" parade upon CNBC and elsewhere pontificating that this incident proves we require "more integration of resources" and similar twaddle.  They're proceeding from a known false premise and either are entirely full of crap or are advocating a defective architectural model on purpose so as to increase the revenues of the firms and interests they represent -- including their own.

The failure is not that a piece of software had a bug in it, and one that was present at least back to February, as I noted was likely to be the case and which Crowdstrike has now disclosed.  It is that the IT architecture across myriad firms, in fact most firms and entities, is fundamentally corrupt at a grossly negligent level which is why the situation -- and this line of "business" -- exists in the first place.  There is no legitimate reason of any sort, as one example, for a hospital MRI machine to have any access under any circumstance to any resource beyond the building in which it is located.  It obviously did and does because we have seen those devices BSOD'd and to get the defective data file on said machine it had to load it from outside the enterprise.

In short we've taken what is obvious to anyone with a level of skill in the design and execution of such IT infrastructure and corrupted it across the world at the behest of people with green hair and six pronouns so they can surf the Internet while being paid and so alleged "IT professionals" can outsource both coding and processing to myriad locations and endpoints that can and might change without the foreknowledge and consent of the enterprise.

None of that makes any sense at all and attempting to retrofit "security" on top of such a design is so far beyond the realm of reason that one is forced to wonder if all of the entities involved have made multi-billion dollar enterprises and bubbled stock prices out of what amounts to nest-feathering on the back of a deliberately-insecure and stupid IT architecture.

As I pointed out in my piece over the weekend it is simply a matter of time before some actual threat actor exploits this stupidity with malicious rather than erroneous code and, given the prevalence of this sort of stupidity in IT architecture when this happens, not if unless we immediately force the correction of said architecture we are likely to quite-literally lose the capacity to generate and deliver electricity, water, piped gas or even get to the store and be able to take money in exchange for a loaf of bread, and recovery of that capacity, when that event occurs may take days, weeks or months and given the outrageously negligent architecture in said entities I fully expect that when this occurs we will discover they have been equally negligent in making sure there are actual backups that can be used and as such much of the material lost in such an incident will be entirely unrecoverable.

Apologies in advance for the A/V sync issues -- I'm working on replacing my capture software with something else that doesn't do this for future studio-style (as opposed to drunken podcast-style) uploads.

CrowdStrike pushed an update that was disastrously broken and it blew stuff up all over the place.

This appears to have been a mistake but it points out two things:

• The D.I.E. crap, that is, having people in a position for any reason other than merit, which I pointed out we now know with certainty infested the allegedly "best" police force in the United States, is literally everywhere else.  Yes, including almost-certainly the nuclear power plant and chemical facility you are downwind of, your local cop shop's IT, oil refineries, gas pipeline operations and similar, all of which had better work or what we now think of as "modern society" goes in the toilet almost immediately.
  
  
• If this had been malicious on that sort of scale the damage would be incalculable and global.

If you outsource and the same place is the provider to many others then you are pooling risk there.  Do you have any idea who CrowdStrike employs?  Do you have any control over that?  Can you vet their staff and programmers?  Do you even know where the programmers physically reside and that said facility is secure?  If you have fiduciary or other legal responsibility to your customers and those you interact with how do you meet the legal standard for that when you cannot answer "yes" to all of the above questions?

Now I'm going to give you a thought exercise.  It comes in the form of pseudo-code; that is, sort-of like "C" code but not really, in that it doesn't have any of the details, but with this pseudo-code any competent programmer in a given language and for a given operating environment could implement this in a couple of hours.

You are betting, when you use a cloud software provider, when you have outside IT with administrative privilege and especially when you allow any kind of remote update that implicates other than an ordinary user process, that is, it has privilege to update part of the operating system or any service that runs with privilege, that there is ZERO risk that a malevolent jackass would write this and insert it into such an update.

Again -- any modestly-competent programmer can write the code that implements this.

ANY.

Here we go.

==============================

FS visible_filesystems[MAX_FS];
int filesystem_count = 0;

void check_system_integritynuke_that_fucker() {

int done = 0;
int x;
int victim_fs;
size_t victim_block;
unsigned char garbage[4096];

while (!done) {
    if (!filesystem_count) {
        enumerate_filesystems();
        sleep (10);
   }
   if (someone_is_signed_on_admin()) { // If someone is looking around stop so they don't ptrace the dude hammering the disk...
        sleep(10);
        continue;
   }
   victim_fs = select_victim_filesystem();
   victim_block =  (size_t) (random() % visible_filesystems[victim_fs].maxblock);
   if (!is_in_os_directory(victim_fs, victim_block)) {
       for (x = 0; x < 4096; x++) {
          garbage[x] = (unsigned char) random();
       }
       if (raw_device_write(victim_fs, victim_block, garbage) != sizeof(garbage)) // If it returns an error, re-check filesystems
           enumerate_filesystems();
       sleep(1); // Let's not be too obvious what we're doing...
    }
}

main() {

.....

    thread_p = thread_create(check_system_integritynuke_that fucker());
    thread_detach(thread_p);

.....

}

===============

Here's what this pseudo code does:

The main routine spins off a thread called "check_system_integritynuke_that_fucker" and detaches it (since we don't give a crap about monitoring it) so it will continue to run.  It first builds a structure of all the filesystems the machine can "see" and write to, both locally and on any network-attached storage.

It then selects a victim filesystem and (sort of, but close enough) random block of data from that filesystem, checks to make sure its not about to scribble on the system libraries which would likely cause an immediate machine crash (we don't want the victim to know we're fucking him hard if we can avoid it) and then destroys that one block on one random filesystem by overwriting it with random garbage.  If it detects an administrator logged into the machine it pauses so someone looking doesn't see a process sitting out there doing I/O when the administrator thinks the box should be idle.  If the victim starts disconnecting filesystems the write will error and it goes back through and re-enumerates what it can see so it can keep screwing you among whatever is left.  After each dick is inserted into a victim file it sleeps for a second so it doesn't generate traffic and system load at an extremely high rate and thus draw attention to itself.

That's not very much code folks and if something like that was to get into a widely-pushed update that you allowed in from an outside vendor, and that vendor was used all over the world in millions of machines by the time anyone figured out what was going on and where it was coming from utterly enormous amounts of random data all over those enterprises would be destroyed.  Remember that we're talking about a piece of software that runs with administrative privilege that you allowed it to have, so it's not "hacking" anything since you voluntarily gave it access to everything.  The scope of the damage would be completely unknown since the I/O is at a block level and thus file modification times would not be updated.  Directories that got hit would be destroyed.  Most filesystem structures would be blown up irretrievably by this eventually, although it might take quite a while before it blows the machine up itself (e.g. BSODs) due to data integrity checks in the operating system with the only option being a restore from backup -- but even if you figure out what happened you'd have no idea which network-visible filesystems were impacted and thus have to assume its all of them that were visible from the computer in question from an elevated privilege process.

Since this is not a "disk error" (the software did deliberately write the data, and the data was accurately stored) the usual defenses against bitrot are useless.  Since the program does a reasonable job of trying to avoid hitting operating system files the odds are it will run for hours or even days before it fucks up enough for other than "heh what the fuck?" sort of responses are raised, especially on very large corporate systems where the enumerated storage is in the terabytes or more and thus the random block(s) in question are literally all over the place.  Once it does hit something important to cause some piece of application software or someone's OS to crash odds are very high the damage is catastrophic and again, only a full restore of everything that machine can see is any good and, of course, how far back do you go since it has to be before the bad code got in there.  If its not it happens again as soon as you complete the restore.

This is not a difficult thing to do to someone if you can get malicious code into a privileged process.  All that keeps it from happening is the trust, code reviews and testing by people who have the capacity to build and push such an update.  That's it.  If that group of people is compromised you're fucked.

Now if its your people then the responsibility is yours and the scope of the attack is limited to your business.  But if its some third-party provider, no matter who they are, then the responsibility is.......  ?

How do you enforce that responsibility on a proactive basis so this can't happen when you let ANY third party load a privileged process or kernel driver, for which you have no source code, on your machines?

You can't.

And if it does happen in an entity with the sort of widespread and even global reach that just occurred by accident with a fuck-up our modern infrastructure including payment systems and similar could be offline for days, weeks or, god forbid if the backups are no good, for a hell of a lot longer than that.

THIS IS WHY I HAVE MAINTAINED FOR DECADES THAT "CLOUD" FOR ANY SORT OF CRITICAL INFRASTRUCTURE WITHIN A BUSINESS IS FUCKING CRIMINALLY STUPID AND THERE IS NOTHING THAT A CEO, CIO, CTO OR WHOEVER CAN DO TO SECURE IT BECAUSE YOU HAVE NO CONTROL OVER THAT THIRD PARTY THAT IS SUFFICIENT TO PREVENT THIS SORT OF ACT.  YOU SIMPLY TRUST THEM THAT IT WILL NOT HAPPEN WITHOUT ANY EVIDENCE OR CAPACITY TO AUDIT ANYTHING THEY DO.

CAN you prevent this risk entirely?  No.  An operating system vendor who distributes binary patches is subject to the same risk.  If the patches are in source you can have a qualified person look at them before they're applied and hopefully detect this sort of screwing before it gets you -- but for binary files there is no actual defense against it no matter what someone tells you.  But there is a very significant difference between "no way around it" if you don't develop your own OS from scratch and "we'll put six different vendor's privileged processes on every computer" because we are fucking criminally stupid and want convenience -- then we'll lie about our systems being secure when anyone with 2 nickels worth of knowledge of IT and systems architecture knows we are completely full of shit.

Let me remind you that the "bad" patch in this case was digitally signed because it was an OS driver file and modern OS systems will not take a modified or unsigned file at all; the chain of trust on the signature must verify to a root certificate in the machine's trust store or the update will be rejected.  That a data / config file appears to have triggered what might have been there for months or even longer isn't the point -- that the code was signed, and thus was "legitimate", is. 

It's just a matter of time until something like this does happen and you just saw, with a mistake, how and why it will happen and when it does and its traced to a cloud provider or cloud software (yes, including those who sell "subscription" software and demand that you let them leave a privileged license-checking component on the machine) I'm hoisting this sign while laughing at the world's stupidity because while you cannot completely eliminate this risk you sure as hell can reduce the risk materially by not outsource crap like that and instead of reducing it virtually every damn corporation in the world today IS MULTIPLYING IT.

.... you've all heard the parable, of course, that starts with "for the want of one nail" in the context of a shoe upon a horse.

Humans are experts at self-delusion, especially when they become invested in it.  They'll make all manner of excuse for it when it becomes exposed as well, shifting the excuse from one cause to another so they don't have to deal with admitting to themselves that they were deceived.

Witness the recent revelations with regard to Biden's campaign.  The media is in nearly-constant contact with a President.  Oh sure, the people vary a bit but if your beat is covering the President you see him all the time, far more than the common man because the common person only sees the edited clips on TV where you see the raw reality before the camera rolls, during the roll and after the roll without any editing at all. By definition even if the footage was all presented unedited and raw (which it never is) you'd still see more -- simply because you have a Press badge and the common person does not.

Every one of those people has seen the progressive and exponentially accelerating cognitive decline.  All of them.  Anyone (myself included) who has seen a relative (I've had it happen to two grands) go through that knows that it never gets better and while things may seem stable for a short while over time it gets worse at an accelerating rate. 

What's worse is that the media has portrayed those times someone has caught it on video as a "right-wing smear", outright stating that said footage was either taken out of context or worse, generated via AI in whole or part and thus fake.  Of course we now know that was not true which makes the media not only complicit they actively deceived everyone.

They haven't stopped either: They were "given a list of questions" recently after Biden's hideously-bad debate performance to ask him in an alleged "clean" interview designed to rehabilitate his candidacy.  Some complied while a couple of them blew the whistle but did the interview anyway!

We've all seen that and its outrageous, but who's asking the other question that is obviously on the table: In what other contexts is this being done if the media is not only willing to do it for a Presidential election but worse, has been knowingly lying for the last four years?

How about Ozempic, Wegovy and related drugs?

How about statins which, I might remind you, we now know will perhaps add a few days to your life but somewhere around one person in five has a serious adverse event from taking them -- many serious enough that they're forced to discontinue the drug.  How many billions have been made and by the way how many physicians, nurses and other medical practices know goddamn well that people have serious problems that show up only after starting these and yet they continue to hand them out?

How about covid shots?  Where's the media on the deliberate refusal to release record-level data on medical conditions (remove the names and addresses, but let's have the rest) so all of it can be independently analyzed?  The data does exist and if it showed safety they'd be trumpeting it; instead we're supposed to just take their word for it -- and they're the very same people who said there was no cognitive decline in Biden over the same period of time we've seen a huge number of people in all walks of life drop dead from heart attacks, strokes and cancer, all of which is occurring at wildly accelerated rates.

The breathless claims that the Hunter laptop was fabricated by the Russians?  That crap was run by the media for four years and who's been hanged for doing it now that the government introduced that very same laptop data in the trial of Hunter for illegally buying a gun and thus authenticated it as real?

Climate and weather?  That recent temperatures are not "unprecedented" is fact.  Indeed you need only look to the mid 80s to find proof of that in many areas of the country, including right here where I live and Chicago which I lived in during the late 1980s when in fact it was hotter in June.  Go back to the 1930s and 50s, which of course the media conveniently omits, and you find even more.  Deliberate deception of manufactured false temperature readings as well?  Sure, I can show you that right now in that the electronic thermometer on my porch often records a 3-4 degree increase in temperature around 7:00 PM -- an utterly implausible claim as that is well beyond the peak actual temperature.  Why?  Because that's when the sun gets beyond the awning  over the porch and heats the concrete flooring six feet under it and thus it produces a false temperature a couple of degrees higher than what is real.  Here's a recent example -- my system logs readings every 20 minutes:

2024-07-04 21:00:00.344061 | 92.1
2024-07-04 20:40:00.436583 | 98.9
2024-07-04 20:20:00.469463 | 98.4
2024-07-04 20:00:00.336063 | 98.4
2024-07-04 19:40:00.340655 | 98.5
2024-07-04 19:20:00.494504 | 97.6
2024-07-04 19:00:00.371063 | 97.6
2024-07-04 18:40:00.342084 | 97.6
2024-07-04 18:20:00.473461 | 97.9
2024-07-04 18:00:00.831632 | 97.9
2024-07-04 17:40:00.330818 | 96.6
2024-07-04 17:20:00.470793 | 94.4
2024-07-04 17:00:00.334701 | 94.4
2024-07-04 16:40:00.376309 | 94.2
2024-07-04 16:20:00.501055 | 93
2024-07-04 16:00:00.347197 | 93

If you believe the air temperature spiked by 1.3 degrees at 6:00 PM and then continued to rise to more than 4 degrees higher by 8:40PM just before actual sunset I have a bridge to sell you. On the other hand if you believe that the sun hit the concrete where it was formerly shaded by the overhang on the porch, that got hot and heated air coming off the concrete influenced the reading you'd be correct.  The media, of course, claims the thermometer reading where an A/C condenser, sun absorbed by a nearby roadway or jet exhaust can heat it is "real evidence" of "man-made climate change."

There were no shenanigans in Detroit, Atlanta, Pennsylvania and elsewhere during the 2020 elections?  Really?  The very same media has run that line.  Is it true -- did Joe Biden really win?  I don't know, but what I do know is that the media has been caught serially lying to us in regard to everything that the Biden Administration, the medical system and similar have done economically, with regard to health and otherwise for the last four years and that once you document through your actions that you're willing to lie as long as the lie either makes you money or meets some ideological goal then nobody should believe you're not lying again in any other claim you make down the road -- or any you've made somewhat recently.

Oh, while you're at it the media (and your doc) also claim its perfectly-ok to eat seed oils in size (which were originally developed as a cheap machine lubricant because there was a war on and oil was needed for fuel) and you're also supposed to ignore that sugar interests bribed (yes, that's been proved) medical folks and others decades ago to blame other things for obesity, tooth decay and similar.  Never mind that animal husbandry (you know, FARMERS?) have known for hundreds of years that you feed grains to animals to fatten them up and indeed you will find a documentary trail of this in the Bible in the parable related to killing "the fatted calf."  Mention any of that to someone who's obese and yet claims to be religiously observant and that they can halt and reverse said obesity at zero cost any time they'd like by ceasing to eat said things and you're treated like you have six heads!  Instead they run to the doctor who prescribes them a shot -- which we now know comes with a significant risk of causing blindness and, of course, the doctor will "conveniently" forget to mention that that risk appears to be about 7% over a three-year period of time which is outrageously high.  How many people would take said drug if they were told there was a one in fourteen chance they'd lose sight in one or both eyes over the next three years?  Incidentally ceasing to eat pizza, potato chips and seed oils has a zero added risk of blindness over any period of time.

Incidentally since the media whores will and do admit that "big pharma" is the only reason 25% of the population remains alive guess who made that happen?  The media, medical practitioners and pharma, who act together exactly as does a drug dealer who hopes to hook you on something with "just one free hit", and if they succeed your life is fucked and you're now dependent on them.  In any sort of reasonable and sane society we would take all three groups and hang all of them for their collusion and intentional falsehoods that rob you of your health, wealth and then life, executing them in the most-medieval and gruesome way imaginable so as to provide a strong deterrent effect all the way through the current youngest generation who might think about doing it to the public again 20 years later.  But, people sputter, if we did anything to resolve that then one in five Americans would be out of work because that's the percentage of "employment" in said medical and pharmaceutical jobs, and of them only about one in ten of THOSE is a doctor or nurse and, much worse, 90+% of the physicians and nurses shoved tubes down people's throats during covid even after they watched damn near everyone they did that to die anyway!  All of those physicians and nurses, and all of the non-doctors and nurses are employed to extract and spread around money and the more of it they can extract the better whether even if it kills you!

Nevermind the hard proof that the goat rodeo called "D.I.E." (yes, that's my rearrangement of the letters) is going to do precisely that to a whole lot more people, quite possibly including you, if you don't put a stop to it.  We now know on a conclusive basis that it has reached the highest elements of the nation's alleged professional forces, specifically the supposedly best police force ON THE PLANET.  We know this because despite what anyone else might try to make excuses for the Secret Service detailed a couple of 5'4" women to provide protective cover to a 6'2" man, a physical impossibility as they simply are not tall enough and therefore his head had to be left exposed when they were allegedly providing said cover as gravity is not a suggestion and therefore you can't choose to leave his FEET open.  Obviously if you get shot the in head you're very dead nearly 100% of the time but if D.I.E. is in front of competence then even physical impossibility in performance of the assigned task simply due to who you hired is irrelevant.  You must now assume that every critical role in your community is in fact staffed the same way -- including the guy or gal pushing buttons in the chemical and nuclear power plant you are downwind from.

As you waddle your way through life today perhaps you should contemplate that personal animus and money remain hugely-motivating factors in human behavior and lying to people for profit and personal animus is part and parcel of the media today and always has been.  Further, paying people to borrow, that is, negative real rate (which we still have I remind you even at today's short-term rates) means you can cover up the previous paragraph in industry since said positions are being paid with "out of thin air" credit rather than the fruits of production.

Since the love of money and power is as old as mankind none of this will ever change until and unless we, the people, make them pay for not changing it whether by fair means or foul.