More Stupid In Corporate America
The Market Ticker - Commentary on The Capital Markets
2017-06-27 13:21 by Karl Denninger
in Technology , 188 references Ignore this thread
More Stupid In Corporate America
[Comments enabled]  

We have yet another "ransomware" game going on globally.

Let me point out that this is driven by idiocy in corporate America.

Ransomware attacks only work because the computers in question are not properly backed up, they do not have a decent plan to keep data safe, they are interlinked including the ability for data to be corrupted "on storage" (such as "in the cloud") and the entire bubble economy in tech is based on more, not less, of this.

Run your "backups" to the cloud?  That's nice -- how fast is your connection and their throughput if you have to restore every machine in the office?  Will it take hours, days or weeks?

That assumes it works too.  When was the last time you verified that?

I know the answer for myself: I verified that my backup system worked this week.  How?  I ran a restore to my laptop.  It took 20 minutes from a cold start off a USB key to being back where it was on a "new" disk.  I didn't do this "in extremis", I did it as part of my regular prove-up that the infrastructure I constructed still works.

That's what competent IT departments do.

Further, I have multiple backup versions available, so if I have to go back to an earlier copy due to some sort of infection I can.  I can also go back to just before the bad thing happened, copy off any changes somewhere else, then restore the earlier one if I have reason to suspect there's a latent problem and then layer over that.

On my servers I can literally go back to a snapshot taken a few hours ago, a few days ago, or a few months ago at any time within seconds.  A full restore of those systems takes quite a bit longer just due to the size of the data store involved but provided the hardware is ok I can revert on a snapshot basis in seconds, making a corrupt file or even entire corrupt filesystem a minor, no-big-deal annoyance.

If you have me install infrastructure in your office I can put that same capability on your network.  Now a "ransomware" attack means nearly nothing other than a (moderately severe) annoyance since once we determine in which hour and which day it hit I can simply revert the snapshot to the one prior to that time and your files are all back to where they were before they got encrypted.  Then we restore the system in question from load media and you're back in business inside of 20 minutes.  Yes, the file you were editing at that moment in time is destroyed, but everything else is fine. We then talk about the recommendation to ****can the idiot who was downloading porno or whatever on his work computer and got infected by doing so, and yes, I can usually figure out who it was.

If your enterprise cannot do this then your IT people have traded off your corporate data security for some ****headed "buzzword" like "cloud."

For this they should be stuffed in the career wood chipper -- feet first -- and if your firm is publicly traded it should be a zero as should those public companies that have advanced and promoted such stupidity.

How many times does this have to happen before the stupid stops?

Apparently the answer is "at least once more."

Go to responses (registration required to post)
 
Main Navigation
MUST-READ Selection:
A One-Sentence Bill To Force The Health-Care Issue

Full-Text Search & Archives
Archive Access

Legal Disclaimer

The content on this site is provided without any warranty, express or implied. All opinions expressed on this site are those of the author and may contain errors or omissions.

NO MATERIAL HERE CONSTITUTES "INVESTMENT ADVICE" NOR IS IT A RECOMMENDATION TO BUY OR SELL ANY FINANCIAL INSTRUMENT, INCLUDING BUT NOT LIMITED TO STOCKS, OPTIONS, BONDS OR FUTURES.

The author may have a position in any company or security mentioned herein. Actions you undertake as a consequence of any analysis, opinion or advertisement on this site are your sole responsibility.

Market charts, when present, used with permission of TD Ameritrade/ThinkOrSwim Inc. Neither TD Ameritrade or ThinkOrSwim have reviewed, approved or disapproved any content herein.

The Market Ticker content may be sent unmodified to lawmakers via print or electronic means or excerpted online for non-commercial purposes provided full attribution is given and the original article source is linked to. Please contact Karl Denninger for reprint permission in other media, to republish full articles, or for any commercial use (which includes any site where advertising is displayed.)

Submissions or tips on matters of economic or political interest may be sent "over the transom" to The Editor at any time. To be considered for publication your submission must include full and correct contact information and be related to an economic or political matter of the day. All submissions become the property of The Market Ticker.

 
Comments.......
User: Not logged on
Login Register Top Blog Top Blog Topics FAQ
Showing Page 1 of 2  First12Last
User Info More Stupid In Corporate America in forum [Market-Ticker]
Karefree
Posts: 1855
Incept: 2008-10-15

Santa Cruz, CA
Report This As A Bad Post Add To Your Ignored User List
I've spent half of my 20+ years in IT in Operations and in all of that time, I have only seen one company actually "practice" a production failover. Database, middleware/web services, applications - the whole thing was failed over to an active/passive HA/DR system. It took more than six hours (and the system was DOWN during this time and not processing orders or any business related transactions) and was considered a success.

lol

----------
And there comes a time when one must take a position that is neither safe, nor politic, nor popular, but he must do it because Conscience tells him it is right.
- Martin Luther King, Jr.
Tickerguy
Posts: 149209
Incept: 2007-06-26
A True American Patriot!
Report This As A Bad Post Add To Your Ignored User List
smiley

That is considered "success"?

smiley

----------
Winding it down.

Karefree
Posts: 1855
Incept: 2008-10-15

Santa Cruz, CA
Report This As A Bad Post Add To Your Ignored User List
I know, right.... it's hysterical!

lol

----------
And there comes a time when one must take a position that is neither safe, nor politic, nor popular, but he must do it because Conscience tells him it is right.
- Martin Luther King, Jr.
Tickerguy
Posts: 149209
Incept: 2007-06-26
A True American Patriot!
Report This As A Bad Post Add To Your Ignored User List
Holy **** I've had a couple of "real, no bull****" failures in the time of running the Ticker. Some of them you never even SAW. ONCE (double-drive hardware fault that ate the raid array) I made the decision to restore "in-place" rather than cut over to the failover system here in Niceville, which resulted in a ~6 hour downtime window while the copy-back took place, ONLY because at the time the load was high enough that if I had not done it that way the performance impairment would have gone on for DAYS while the copy-back from the fail-over system to the main took place at lower priority, and I was able to get the hardware in question replaced inside of an hour.

So that was an intentional downtime decision as opposed to a "aw ****" sort of circumstance. In point of fact actually moving the DNS pointers and telling DBMS to swap over was a "couple of minute" deal if I had decided to go that direction instead, and if the main site had been *destroyed* (or effectively so) that would have been the path taken.

----------
Winding it down.
Dennisglover
Posts: 686
Incept: 2012-12-05

Huntsville, AL
Report This As A Bad Post Add To Your Ignored User List
At a former employer I had to walk into a development lab one day and tell the lead developer that his team's literal million man-hours-of-work was gone from his main development server, and there was nothing on the backup tapes. That might have made my insides a bit bothered.

As it turns out, though, I had told him years before then that neither my employer nor the government agency who gave us the contract would allow any backup/restore capability testing because of the "production schedule". Since that conversation he required all of his people (including himself) to maintain hourly-to-daily backups on at least one remote system every workday. So he took it in stride, waited until SGI replaced the lost disk set, and restored his files. Downtime about 36 hours, and not truly catastrophic.

Then he decided to get in on the act, and I was able to design at least an automated backup/restore solution that ran unattended to IBM 3494 libraries in the facility. But even he wasn't able to force the company and the government to make it anything like a robust backup system. So with the backup servers and libraries in the same server room as the 500 or so servers and workstations (all diskful) a tornado would have been a true catastrophe. More, there was no allowance for off-site storage of backup sets, which were simply left in the libraries all the time.

As for testing? My manager told me to copy a bunch of files to a chosen quiescent system on the network, force a backup, blow the DAR files away, and see if I could restore the files to the original and a separate system! I asked how many times he wanted me to do that, and he said once would prove that it works.

A few years later that manager fired me. No great loss.

----------
TANSTAAFL
Kwaldman
Posts: 5
Incept: 2010-06-10

Canton MA
Report This As A Bad Post Add To Your Ignored User List
looks like there was a ransonware on Linux last week - where the hacker got a $1.6M payday. http://blog.trendmicro.com/trendlabs-sec....

That has to sting
Nomullet
Posts: 7744
Incept: 2007-11-11

SW
Report This As A Bad Post Add To Your Ignored User List
Karl, 'the cloud' is old hat now the new thing is 'fog computing' (I wish I was kidding)

----------
A bad day of programming is better than a good day of management.
Tickerguy
Posts: 149209
Incept: 2007-06-26
A True American Patriot!
Report This As A Bad Post Add To Your Ignored User List
"Foggy" your data security is......

----------
Winding it down.
Aztrader
Posts: 7796
Incept: 2007-09-10

Scottsdale, AZ
Report This As A Bad Post Add To Your Ignored User List
FEDEX site won't come up. Something amiss...............
Gantww
Posts: 1410
Incept: 2011-04-22

Nashville, TN
Report This As A Bad Post Add To Your Ignored User List
You know, we should find some people on 4chan and get them to push the idea of "fart computing". You know the buzzword bull**** bingo folks would pick it up and push it through the tech press in like five minutes. They've done it in other areas where the bull**** piled too deep (see free bleeding, and all the stuff about Pepe in the last election), so it might be fun. Someone clever could probably put a pitch deck together for it that would look professional.

Fart computing. It's like a transient version of the cloud, so you never have to worry about attacks on data at rest....

----------
Pissing on the host in the middle of the living room with guests present is a pretty good reason for the host to forcibly remove one from the scene, in my humble and correct opinion. - Jack_Crabb
Ahhz
Posts: 227
Incept: 2011-06-12

Report This As A Bad Post Add To Your Ignored User List
Tickerguy wrote..
"Foggy" your data security is......

Indicates excrement between ears it does...


(Yoda meme for the day from the Ticker forum. Not directed at Nomullet personally at all, just those people who think such things are a grand idea.)

Unknownsailor
Posts: 397
Incept: 2009-04-06

Bremerton, WA
Report This As A Bad Post Add To Your Ignored User List
Seems to me that asking a prospective employer about their back-up procedures could be a nice go/no go gate to accepting an offer...
Tickerguy
Posts: 149209
Incept: 2007-06-26
A True American Patriot!
Report This As A Bad Post Add To Your Ignored User List
These days they would all flunk

----------
Winding it down.
Maynard
Posts: 348
Incept: 2007-11-27

Like many PNW
Report This As A Bad Post Add To Your Ignored User List
I am just one dude (and family) doing nothing really mission critical but I have 4-5 backup drives in different locations and travel with 2. Granted they aren't all synced all the time but anything important is really backed up. With storage as cheap as it is there is no excuse to loose data to hackers or hardware fails.
Elkad
Posts: 289
Incept: 2009-09-04

Report This As A Bad Post Add To Your Ignored User List
I'll be shocked if one of my clients doesn't get hit by this current round. I just don't know which yet.

When they refuse to update antique servers (some are still running 2003), and have medical billing coders in India connecting on open port RDP sessions, it's going to happen.

I setup VPNs. It's too complicated for their remote users to click 2 icons instead of one. Or their ISP in India is blocking fragmented packets so it won't connect. So they tell me to leave the RDP ports open. I counsel them against it. In writing. And check the backups. Best I can do.
Tickerguy
Posts: 149209
Incept: 2007-06-26
A True American Patriot!
Report This As A Bad Post Add To Your Ignored User List
The frag block issue is one that pisses me off BIG.

----------
Winding it down.
Tinman
Posts: 282
Incept: 2008-02-16

Report This As A Bad Post Add To Your Ignored User List
Fortune 50 company here... we used to do yearly full DR tests to fully restore a data center within 12 hours. We have not done that in ten years. They froze the pension fund so who ****ing cares, not me.
Dennisglover
Posts: 686
Incept: 2012-12-05

Huntsville, AL
Report This As A Bad Post Add To Your Ignored User List
Soon after the formerly unmentioned-by-name employer fired me I had occasion to interview with a rather large Northeastern "financial services company", whose revenue is now listed at North of $0.5T. Their concern, after 9/11/2001, was that having a single data center in a certain East Coast metropolis (where the New Year's Eve ball drops every year), they wanted to "decentralize" and stuff.

Their idea was that they would move it all to Charlotte and "collocate" in Denver!

I asked what good it would do to place peer data and backup centers in both a huge banking center and a huge transport center. If gargantuan infrastructure was likely to be hit by a terrorist attack, didn't they think those places would be hit, and almost certainly? The interviewer did give me a chance to offer my alternative.

I told him that there were many small towns, abandoned telephone company COs, all over the country, and (because of the growth of actual ISPs who were not "just telephone companies") there were lots of non-descript little gray and tan concrete block buildings either empty then or soon to be empty, and they ought to be thinking about creating dozens or hundreds of "collocated sites", so they could in fact de-centralize. (Of course, my idea was that if you don't advertise what you're doing, and you don't put out all kinds of news releases about it, then you're lots less likely to be "found out". Silly me, I guess.)

I'd guess it was too much for them to think about, or they thought it cost too much, so that's why I never heard from them again.

It was $200B+ in 2003, now it's $500B+. I still don't believe they even imagine it can all disappear in a literal instant.

----------
TANSTAAFL
Gable
Posts: 719
Incept: 2009-07-04

Retired in NC Mountains
Report This As A Bad Post Add To Your Ignored User List
Quote:
I still don't believe they even imagine it can all disappear in a literal instant.


I worked for HP as a mass storage specialists. After 9/11 is was widely quoted by our sales folks that 95+% of companies in the World Trade Center that did not have off site backups went out of business. I am not sure how accurate it was or if it had more to do with their employees passing away, but we sold a huge amount of mass storage and replicating software and cluster servers to lots of customers.

What I would like to know is just how vulnerable our power grids really are. I have read all kinds of worse case scenarios, but there is no way I can judge their accuracy. Any TFers have any real world experience with that industry or how vulnerable they really are???


----------
In all of history, no government became more honest, less corrupt, or respected its citizens' rights more as it grew in size. E.L. 2016
Mj71
Posts: 136
Incept: 2009-03-14

Report This As A Bad Post Add To Your Ignored User List
Don't ignore the importance of 'air-gapping' your backups. If your backup location is just another drive letter, you're fooked.
Snowman
Posts: 1929
Incept: 2009-03-09

avoiding yellow snow
Report This As A Bad Post Add To Your Ignored User List
Dennisglover: I'm guessing Cathy Bessant's shop in Charlotte. BAML has about 40 data centers in the US, another 20 or so outside, running about 5,000 applications (many legacy duplicates from ML, LaSalle etc), and are piggy-backing on other WS bank's ops centers in New Jersey. Every third employe at the bank works for her. Her shop is no different that the other TBTF. JPMC is even more complicated, not to mention HSBC (probably the least integrated of them all). Bank GT&O departments are enormous sinkholes. Absolutely hugely complex global operations. I am always amazed that these banks can even operate in the first place. Forgot about trading, settlement, credit and regular banking risks. IT risks are top of the list. Banks are IT and data companies first, and they do customer stuff on the side, though they behave as it were the opposite.
btw, I think you meant assets, not revenue.
Jdough
Posts: 80
Incept: 2012-05-04

The Lone Star State
Report This As A Bad Post Add To Your Ignored User List
Where I am they do have offsite backups and have successfully restored from them, usually due to human error problems (DELETE FROM MAIN_USER_TABLE type stuff). The DR is hilarious though, a bunch of "plans" to move everything to alternate data centers and the test is to review the plans.

----------
The federal head will possess, without limitation, almost every species of power that can, in its exercise,
tend to change the government, or to endanger liberty; the people will have but the shadow of representation, and but the shadow of security for their rights and liberties
Dennisglover
Posts: 686
Incept: 2012-12-05

Huntsville, AL
Report This As A Bad Post Add To Your Ignored User List
Snowman--Right as rain, I did mean to write "assets". Thank you.

----------
TANSTAAFL
Clay3482
Posts: 447
Incept: 2008-11-13

Alaska
Report This As A Bad Post Add To Your Ignored User List
I am sure (just speculating here) that at least one company has not recovered from the last ransomware attack.

You know - they might have got the network back up after 3 weeks - But I bet to this day it is REAL SLOW - like 30 minutes from password to being logged on slow.

I bet if this speculated company does exist it still has no printer support, email, or needed stuff like that. I bet their new plan is to buy all new computers for everyone because the current system is just NOT FIXABLE. I bet they have also advised employees to use GMAIL and Google APPS because I AM SURE that will fix the problem.
Login Register Top Blog Top Blog Topics FAQ
Showing Page 1 of 2  First12Last