On Sandy And Foolishness
The Market Ticker ® - Commentary on The Capital Markets
Posted 2012-10-30 09:40
by Karl Denninger
in Editorial
Ignore this thread
On Sandy And Foolishness
 

So how silly are people feeling this morning?

Last night one of the more-amusing sights was watching NYC go dark as Sandy came ashore.

As someone who lives in a hurricane zone I'm no stranger to mass blackouts.  I'm also no stranger to this thing called "a generator", a miraculous mechanical device that is designed to work when the utility power doesn't.

There were multiple reports last night from allegedly-generator equipped locations such as hospitals asking for some rather odd things -- like jerry cans full of fuel along with multiple reports of "backup power failures."  The chuckle factor went off-scale with me when I started hearing of those reports for a whole host of reasons.

There ought to be a mass of firings among IT and facilities management folks this morning in NYC.  The entire point of installing a backup generator of any sort is to make sure that when the utility fails irrespective of the reason you still have electrical power.  Some people think that this means buying a big generator, installing it, and then you're all set.

Uh, nope.

First, you have to consider where the generator is to be located.  You can't put it where any of the foreseeable perils will destroy it (e.g. where it can be flooded out!) or you just wasted your money.  Likewise the fuel tank for it, which is subject to various code requirements and has to be installed and vented in such a fashion that's secure as well.  Note that fuel is lighter than water; as such a fuel tank (especially one partially full) floats, so securing it against flooding is in fact quite a trick (but not impossible.)  Then there's the matter of getting the fuel from the tank to the generator; if your fuel pump floods out you have a further problem.

So now you have a running generator, or so you think.  Now about that power; the switchgear (contactors, breaker panels, etc) between the power source and the load also must be secure against all these perils.  Needless to say having all your switchgear in a sub-basement where it can fill up with water could be a problem.

When it comes to computers, chillers in big buildings are not quite like an air conditioner in a house.  They require water for their heat exchanger, and the unit that ultimately dissipates the heat to the atmosphere is (usually) on the roof.  If there's no water flow to your chillers in the computer room and/or no fan on the roof unit that's running, your wonderfully-climate controlled computer room no longer has air conditioning!  If you've ever been in such a room when all the roaring fans go silent there's this pregnant pause among the people there who know full-well that unless you can get those chillers back online in a hurry the temperature in that closed space will rise to the point that there is no choice other than to shut everything down.

Yes, this means you need to make sure that the circulation pump is safe (along with its power) and that the heat you reject can actually be dumped (which means you need to know that every part of the system from your chiller onward to that point is both secure against the perils and can be powered.)

Oh, and it helps if you test all this too, so when you actually need to use these fancy systems "in anger" you don't wind up with what happened in at least one instance last night where NYFD had reports of "smoke in the generator room" (e.g. all that nice planning was for naught as the generator caught fire when they attempted to use it!)

So how is it again that we have all these fancy-dancy office buildings, not to mention critical infrastructure such as hospitals, that didn't manage to keep themselves lit during the storm?

Yeah, facilities managers are real smart these days (snort!)

Discussion below (registration required to post)
 

Main Navigation
Full-Text Search & Archives
Archive Access
Get Adobe Flash player





Blogtalk 3:30 CT Mondays
Items To Look At


Discuss The Capital Markets along with daily technical analysis with our Gold Donor program.

Where We Are, Where We're Heading (2013) - The annual 2013 Ticker

Links and Blogroll
Our policy on reciprocal links: Send us an email with your information and why you think your blog or news site would make a good addition - in most cases reciprocal link requests will be granted.
Legal Disclaimer

The content on this site is provided without any warranty, express or implied. All opinions expressed on this site are those of the author and may contain errors or omissions.

NO MATERIAL HERE CONSTITUTES "INVESTMENT ADVICE" NOR IS IT A RECOMMENDATION TO BUY OR SELL ANY FINANCIAL INSTRUMENT, INCLUDING BUT NOT LIMITED TO STOCKS, OPTIONS, BONDS OR FUTURES.

The author may have a position in any company or security mentioned herein. Actions you undertake as a consequence of any analysis, opinion or advertisement on this site are your sole responsibility.

Looking for "The Best of Market Ticker"? Check out
Ticker Classics.

Visit the forum to discuss this and other investing-related topics; see the FAQ on the forum for information about Gold Donor status including access to our technical analysis video server.

Market charts, when present, used with permission of TD Ameritrade/ThinkOrSwim Inc. Neither TD Ameritrade or ThinkOrSwim have reviewed, approved or disapproved any content herein.

The Market Ticker content may be reproduced or excerpted online for non-commercial purposes provided full attribution is given and the original article source is linked to. Please contact Karl Denninger for reprint permission in other media or for commercial use.

Submissions may be sent "over the transom" to The Editor at any time. To be considered for publication your submission must include full and correct contact information and be related to an economic or political matter of the day. All submissions become the property of The Market Ticker.

Leads on stories of current economic and political interest are always welcome. Our fax tip line is 850-897-9364; please include contact information with your transmission.

 
Comments.......
User: Not logged on
Login Register Top Blog Top Blog Topics FAQ
Showing Page 1 of 2  First12Last
User Info On Sandy And Foolishness in forum [Market-Ticker]
Vitchilo
Posts: 4977
Incept: 2011-04-27

Report This As A Bad Post Add To Your Ignored User List
Indeed Karl... but you know people... they think they are above everyone else, that nothing can touch them... and then they end up in a bad position... and say ``nobody could have foreseen this!`` as an excuse...

Anyway, to those asking if the election could be canceled/delayed... it has NEVER been done... not even during the civil war... congress would need to do a special session and put a new date... which is very unlikely.

So election is still a go on November 6... unless Obama abuses his powers once again and uses an executive order.

----------
We hang the petty thieves and promote the great ones to office.
Digitlman
Posts: 351
Incept: 2011-03-04

Report This As A Bad Post Add To Your Ignored User List
Looking at a photo of NYC from early this morning and they only building lit like a Christmas tree? Goldman Sachs.

As much as I would like them to crumble into dust, their facilities people seem to know what they are doing....
Peterm99
Posts: 5176
Incept: 2009-03-21
Gold
SoCal
Report This As A Bad Post Add To Your Ignored User List
Often, the locality requires permits for installation of systems like this, post-installation inspections, and quite frequently, periodic inspection by code enforcement personnel.

Thus, these failures also serve as still more examples of gov't incompetence by imposing bureaucratic requirements that are often useless for anything other than to increase gov't employee headcount.

----------
". . . the Constitution has died, the economy welters in irreversible decline, we have perpetual war, all power lies in the hands of the executive, the police are supreme, and a surveillance beyond Orwell’s imaginings falls into place." - Fred Reed
Geschrei
Posts: 490
Incept: 2009-02-23
Gold
Report This As A Bad Post Add To Your Ignored User List
As you know personally, Gen, my work facility has a sophisticated backup power system consisting of a massive diesel generator capable of maintaining the core operations (including air conditioning required for keeping the equipment cool) for roughly 50 hours. In the run up to major storms we also make sure we have additional diesel on hand in case our power is out for more than a couple of days. We test it regularly to make sure it's there when we need it. And to make sure the operation remains uninterrupted, there's an automatic electronic relay that maintains a steady load to the UPS so that service is uninterrupted in the event of loss of external power.

Even so, when Hurricane Dennis hit in 2005, we had made all of the appropriate preparations and were ready to roll when the power failed...except that we didn't anticipate the electrical flow into the building becoming rapidly intermittent...within seconds, the sudden on/off of the power fused the automatic relay in the 'offline' position, so when the electricity failed completely, we could no longer get power from the generator into the building (naturally, this was just as the storm was reaching its peak). So we had a perfectly functional generator with 50 hours of fuel, and still no power.

Needless to say, that system has been made more robust, and a momentary delay programmed into the relay mechanism. But no one thought of that until AFTER the first time we needed it.

----------
“The danger is not that a particular class is unfit to govern. Every class is unfit to govern.”

Lord Acton (1834 - 1902)
Blackswan
Posts: 5589
Incept: 2007-11-06
Gold
Just outside of Philly
Report This As A Bad Post Add To Your Ignored User List
This reminds me of a former employer - Thomson.. now Thomson Reuters. The database center was impressive. They had three levels of redundancy for power backup. Three huge backup generators - routinely tested and maintained.

That is pathetic that a hospital's backup failed especially since they had several days to prepare for a storm.

----------
“It’s checkmate. Everywhere it’s checkmate.”
Hugh Hendry

Otiswild
Posts: 5677
Incept: 2009-03-09
Green
Inside you, the force is!
Report This As A Bad Post Add To Your Ignored User List
Don't forget a number of colo hotels are positioned on the Jersey City coast facing downtown NYC. Smart!!

Also, getting a permit or neighbor permission to run a generator in a NYC apartment or co-op? Good Luck With That!
Bertdilbert
Posts: 2694
Incept: 2008-12-22
Gold
CA
Report This As A Bad Post Add To Your Ignored User List
Not to mention over time, stuff will grow in diesel fuel tanks if an additive is not used. She might fire up and run for a time until the filter clogs,

----------
Dear Euroland: Relax, Germany has a plan for your money!

Political Capital Defined: We are out of money but will tax our citizens for whatever it takes to "SAVE" the Euro.
Genesis
Posts: 131439
Incept: 2007-06-26
Admin A True American Patriot!
Report This As A Bad Post Add To Your Ignored User List
Bert, intelligent folks have a polishing schedule for stored fuel for exactly this reason.

----------
I don't care if it makes sense -- only if it makes money. -- Me
Bank (n): See scam, fraud and theft. Eat a bankster -- they're low-carb.
What part of "shall not be infringed" was unclear?
Wearedoomed
Posts: 3610
Incept: 2009-01-14
Silver
slightly red state
Online
Report This As A Bad Post Add To Your Ignored User List
I have service with a particular VoIP provider that has a solid reputation in some circles. Really, I should say they had a solid reputation. For the past month or so, they've been the victim of a truly nasty and persistent DDoS attack that came at them in several different ways and knocked out their service to a very significant degree. They finally got that mostly straightened away within the past week or so... but it turns out that their one and only PoP is in lower Manhattan, without any significant DR! They're fired as soon as I can port my DIDs elsewhere.

----------
Lies, all of it.

Jpg
Posts: 332
Incept: 2009-03-23

MI
Report This As A Bad Post Add To Your Ignored User List
Quote:
When it comes to computers, chillers in big buildings are not quite like an air conditioner in a house.
There was a time, several decades ago, when I was stationed at an Air Force base near St. Louis.

Had a bad winter storm ... -20F, 30 mph wind.

Froze the chillers for our computer center.

Tried opening the doors, but that didn't really circulate the air adequately.

Called over to the base motor pool to ask for a portable air-conditioning trailer.

Their response was "Yeah, we'll send one right over. As soon as we dig it out from under the snow drift".

As I recall, we eventually got the chillers thawed. Don't recall how long we were "out of operation".

I seem to recall that after the chillers were thawed, they got some antifreeze.

Reason: .
Vernonb
Posts: 433
Incept: 2009-06-03
Gold
East of Sheol
Online
Report This As A Bad Post Add To Your Ignored User List
Quote:
Looking at a photo of NYC from early this morning and they only building lit like a Christmas tree? Goldman Sachs.

As much as I would like them to crumble into dust, their facilities people seem to know what they are doing....


LOL.... forget about help from the neighbors (or lending help)

When you have screwed the world vigilance becomes the primary concern for survival.

----------
"The welfare of the people in particular has always been the alibi of tyrants, and it provides the further advantage of giving the servants of tyranny a good conscience.”
-Alber Camus (1913-1960)
Bertdilbert
Posts: 2694
Incept: 2008-12-22
Gold
CA
Report This As A Bad Post Add To Your Ignored User List
I am getting Google/Youtube errors this morning... Never got them before.

----------
Dear Euroland: Relax, Germany has a plan for your money!

Political Capital Defined: We are out of money but will tax our citizens for whatever it takes to "SAVE" the Euro.
Gable
Posts: 426
Incept: 2009-07-04
Green
Report This As A Bad Post Add To Your Ignored User List
Never ceases to amaze me how people, who consider themselves smarter than the rest of us, keep making really dumb mistakes most of us understand is everyday maintenance and preparedness.

----------
In all of history, no government became more honest, less corrupt, or granted its citizens more rights as it grew in size. E.L. 2011

Ellie's Law-As an online discussion about the failures of the Obama Administration continues, the probability someone shouting "It's Bush's Fault" approaches 1
Sd79
Posts: 3138
Incept: 2008-10-12
Silver
SoCal
Report This As A Bad Post Add To Your Ignored User List
Every hospital I've ever been in and most nursing homes, have weekly gennie tests that last for up to an hour.

I can't believe they faked 'em or screwed it up so bad when it came time for the real deal.


----------
“Everybody is a genius. But if you judge a fish by its ability to climb a tree, it will live its whole life believing that it is stupid.”
~ Albert Einstein
Mrholty
Posts: 1145
Incept: 2007-07-27
Green
Wisconsin
Report This As A Bad Post Add To Your Ignored User List
when NYC had the blackout in what 2002-03 I worked at one of the airports. The screwed up way that is the PANYNJ is some of the Terminals are leased/operated by the tenant. During that outage, everything kicked in correctly and ran for 12 hours. However there was fuel in the regular location and then a backup location. Once it switched to backup we quickly realized that the pump for the backup was electrically powered. 20 year old system, signed off annually by the Port, tested annually, etc for 20 years and missed.

The sad thing about it was it allowed us to run at 40% capacity which is what we wanted. The PANYNJ asked us to shut down as their terminals were closed as their backups didn't work and the other airlines were upset that we were operating (on a limited schedule) but operating where they weren't.
Bagbalm
Posts: 4323
Incept: 2009-03-19
Green
Just North of Detroit
Report This As A Bad Post Add To Your Ignored User List
I used to service a generator that was natural gas fired and started off a flywheel. As soon as line voltage failed it engaged a electromagnetic clutch and switched over. We'd test it monthly. When it was finally needed it did not run. Reason? The building hired a new JANITOR who had access the the equipment rooms. He saw that the gas valve was open to the generator and although it was none of his business and not his job he closed it because he had an irrational fear of gas and felt it was a safety hazard open. Same guy would not answer his office phone if there was a storm. He was full of phobias and we couldn't get him fired.
Sstumpff
Posts: 34
Incept: 2009-08-24

Gilbert, az
Report This As A Bad Post Add To Your Ignored User List
Genesis:

I figured you of all people would recognized the rust of margin compression. The only people who need fired at these places are the CFO's.

----------
I heart Elizabeth Warren
Wearedoomed
Posts: 3610
Incept: 2009-01-14
Silver
slightly red state
Online
Report This As A Bad Post Add To Your Ignored User List
Look at what people at DirectNIC's New Orleans data center did during Katrina. One guy there, a Green Beret, had a blog detailing what they were doing to keep things running. It went viral. If only some companies actually gave a ****...

----------
Lies, all of it.
Jal
Posts: 519
Incept: 2009-03-25

Report This As A Bad Post Add To Your Ignored User List
"If I do it and its not in my job description/responsibility I'll get fired."

Its called being part of the team, being a good team player.

Don't make waves. Don't disturb the dust bunnies.
Ptoemmes
Posts: 143
Incept: 2009-04-13

Report This As A Bad Post Add To Your Ignored User List
This probably belongs in the foolishness category, but that has not stopped me before ;-):
http://www.politico.com/news/stories/101....

=====

Federal law requires presidential elections to be held on the Tuesday after the first Monday in November, but it also provides that if a state “has failed to make a choice on the day prescribed by law, the electors may be appointed on a subsequent day in such a manner as the legislature of such State may direct.

======

Hmmm...so could Sandy affected NE states, in theory, punt elections and have their legislatures choose the electoral college members? What about early/absentee voters?

I no not believe there is a national law - or even a state law - that binds electors to vote based on the popular vote allocation of their state though they may suffer "shunning" if they do not - not sure about that. I believe Maine and Nebraska allocate other than state winner take all.
Andysvw
Posts: 1873
Incept: 2010-06-26
Green
Tujunga Ca
Online
Report This As A Bad Post Add To Your Ignored User List
Every time a disaster strikes like Katrina or the Japan earthquake we get these stories. It makes me sick. In this case the storm did not sneak up on them. They should have tested them over the weekend.

Public hangings? If people die someone should fry. The US Navy takes this to the mat. All of that equipment is proven and tested on a regular basis. A flood should not take out the emergency generator system ever.
Drjerry
Posts: 591
Incept: 2007-11-06
Green
Seattle
Report This As A Bad Post Add To Your Ignored User List
I lived in Canada for a number of years. AKA The Great White North.
It snows in Canada.
Living there you learn to have backups. And backups for the backups.
If the storm/outage lasts longer than the supply of diesel, we have complete heating, cooking and sleeping ready to go. Only difference is we don't have to pitch the tent.
Dazedncornfused
Posts: 324
Incept: 2010-10-13
Green
Report This As A Bad Post Add To Your Ignored User List
'Verizon and other carriers in the area are down as well. Generators are unable to pump fuel due to the flooding in the basements.'

'Elsewhere flooded servers of Datagram Inc in New York City, brought down several media websites it hosts, including Huffington Post and Gawker.'

from http://www.dailymail.co.uk/news/article-....

##########################
Here's a couple of stories about that Green Beret who worked for DirectNIC
http://www.datacenterknowledge.com/archi....
http://www.baselinemag.com/c/a/Business-....

----------
Stand up and be counted or line up and be numbered.
N9lhm
Posts: 92
Incept: 2010-01-28
Gold
South Bend, IN
Report This As A Bad Post Add To Your Ignored User List
I take care of our diesel generators here at work, and do generator repair on the side. While weekly no-load/monthly load tests are a must, like Sd79 said, you also have to realize that all that running a generator for an hour proves is that your generator will run for an hour :) Like Karl and others have said, unthought-of things like fuel supply can get you if you haven't considered them.

Login Register Top Blog Top Blog Topics FAQ
Showing Page 1 of 2  First12Last