Skip to content(if available)orjump to list(if available)

Loose Wire Leads to Blackout, Contact with Francis Scott Key Bridge

psunavy03

Although I was never named to a mishap board, my experience in my prior career in aviation is that the proper way to look at things like this is that while it is valuable to identify and try to fix the ultimate root cause of the mishap, it's also important to keep in mind what we called the "Swiss cheese model."

Basically, the line of causation of the mishap has to pass through a metaphorical block of Swiss cheese, and a mishap only occurs if all the holes in the cheese line up. Otherwise, something happens (planned or otherwise) that allows you to dodge the bullet this time.

Meaning a) it's important to identify places where firebreaks and redundancies can be put in place to guard against failures further upstream, and b) it's important to recognize times when you had a near-miss, and still fix those root causes as well.

Which is why the "retrospectives are useless" crowd spins me up so badly.

stackskipton

>Which is why the "retrospectives are useless" crowd spins me up so badly.

As Ops person, I've said that before and it's mainly because most companies will refuse to listen to the lessons inside of them so why am I wasting time doing this?

To put it aviation terms, I'll write up something being like (Numbers made up) "Hey, V1 for Hornet loaded at 49000 pounds needs to be 160 knots so it needs 10000 feet for takeoff" Well, Sales team comes back and says NAS Norfolk is only 8700ft and customer demands 49000+ loads, we are not losing revenue so quiet Ops nerd!

Then 49000+ Hornet loses an engine, overruns the runway, the fireball I'd said would happen, happens and everyone is SHOCKED, SHOCKED I TELL YOU this is happening.

Except it's software and not aircraft and loss was just some money, maybe, so no one really cares.

drivers99

> it's important to recognize times when you had a near-miss, and still fix those root causes as well.

I mentioned this principal to the traffic engineer when someone almost crashed into me because of a large sign that blocked their view. The engineer looked into it and said the sight lines were within spec, but just barely, so they weren't going to do anything about it. Technically the person who almost hit me could have pulled up to where they had a good view, and looked both ways as they were supposed to, but that is relying on one layer of the cheese to fix a hole in another, to use your analogy.

kennethrc

Likewise with decorative hedges and other gardenwork; your post brought to mind this one hotel I stay regularly where a hedge is high enough and close enough to the exit that you have to nearly pull into the street to see if there's oncoming cars. I've mentioned to the FD that it's gonna get someone hurt one day, yet they've done nothing about it for years now.

astrocat

this is essentially the gist of https://how.complexsystems.fail which has been circulating more with discussions of the recent AWS/Azure/Cloudflare outages.

null

[deleted]

null

[deleted]

tialaramex

Note that "Don't make mistakes" is no more actionable for maintenance of a huge cargo ship than for your 10MLoC software project. A successful safety strategy must assume there will be mistakes and deliver safe outcomes nevertheless.

buildsjets

In a well engineered control system, any single failure will not result in a loss of control over the system.

Was a FMECA (Failure Mode, Effects, and Criticality Analysis) performed on the design prior to implementation in order to find the single points of failure, and identify and mitigate their system level effects?

Evidence at hand suggests "No."

fabian2k

The big problem was that they didn't have the actual fuel pumps running but were using a different pump that was never intended to fulfill this role. And this pump stays off if the power fails for any reason.

The bad contact with the wire was just the trigger, that should have been recoverable had the regular fuel pumps been running.

jtokoph

It’s been noted that automatic failover systems did not kick in due to shortcuts being taken by the company: https://youtu.be/znWl_TuUPp0

DamnInteresting

bmelton

That was super helpful. I was assuming from skimming the text description that it was a failed crimp

A lot of people wildly under-crimp things, but marine vessels not only have nuanced wire requirements, but more stringent crimping requirements that the field at large frustratingly refuses to adhere to despite ABYC and other codes insisting on it

Aurornis

> A lot of people wildly under-crimp things

The good tools will crimp to the proper pressure and make it obvious when it has happened.

Unfortunately the good tools aren't cheap. Even when they are used, some techs will substitute their own ideas of how a crimp should be made when nobody is watching them.

comeonbro

Instant classic destined straight for the top of the Intro to Engineering canon.

Where do you think it will fit on the list?

jojobas

"Contact" is a weird choice of words.

1970-01-01

So there were two big failures: Electrician not doing work to code; inspector just checking the box during the final inspection.

nightpool

No, there was a larger failure: whoever designed the control system such that a single loose wire on a single terminal block (!) could take down the entire steering system for a 91,000 ton ship.

gishh

The date for bridge completion was bumped from 2028 to 2030 already. I assume it won't be done until 2038. It is absolutely murdering traffic in the Baltimore area, not having a bridge. I would be super interested in seeing where every single dollar goes for this project, I assume at least 1/3 of it will be skimmed off the top.