When it Comes to “Big” News Stories, I like to focus on some of the smaller points — the parts that illustrate the “thinking” aspects of the stories, or what should be the “lessons learned” from them. Hawaii’s “ballistic missile” incident is a perfect example. Let’s start with my take on it, from True’s 14 January 2018 issue:
If You Have to Fry, Hawaii is the Place to Do It
The false cell phone warning of “BALLISTIC MISSILE THREAT INBOUND” to Hawaii sent islanders and tourists into a panic — after all, it even said “THIS IS NOT A DRILL.” The real problem, officials say, is that while it was quickly determined to be a false alarm, it took 38 minutes for a follow-up message to be sent through the same warning systems to let everyone know they weren’t about to die. In a press conference later, Gov. David Ige explained how it happened: an “employee pushed the wrong button.” (RC/Honolulu Star-Advertiser) …Not too reassuring: that’s how the missiles would be launched in the first place.
It’s Worse than You Think
But wait, there’s more! How could the guy have pushed “the wrong button”? Because the software interface was garbage. At shift change, the state’s Emergency Management Agency employee taking over the position was supposed to test the warning system. Hawaii has been a bit on edge with the escalation of tensions with North Korea, and Hawaii is an easier target for that country’s despot to hit than the American mainland. Hence the more attention being made to early warning systems in Hawaii.
But here’s how that employee had to do it: he had to use a pull-down menu on his computer and select one of two options: “Test missile alert” and “Missile alert” — and when aiming for the former he actually clicked the latter. Have you ever clicked where you didn’t mean to? I sure have. But “important” selections should be away from “less important” functions, and they were right together.
As far as I can tell, both options did have a “Did you really mean to click this?” confirmation process, which of course trains the operator to ignore what it says since it’s always there, always asking the same thing, so naturally he clicked “OK” on that.
But It Gets Worse
The first scandal is, it was that easy to send a false alert. The second scandal is that there was no similar function to send any sort of retraction: it took so long that both the state EMA and the governor were able to get “tweets” out long before the retraction was sent via the same cell phone emergency message service, 38 minutes later. The reason? While Hawaii’s EMA has standing permission from the Federal Emergency Management Agency to use the cell phone system to send missile alerts, they had no permission to send any other sort of alert, according to Hawaii EMA spokesman Richard Rapoza.
“We had to double back and work with FEMA” to write the correction, and then to get approval to send it, Rapoza said, “and that’s what took time.” He says they have already updated the software to be able to instantly send such a correction alert, but he didn’t mention whether the user interface that created the mistake in the first place has been addressed.
“Part of the problem was it was too easy — for anyone — to make such a big mistake,” Rapoza said.
That’s not “part” of the problem, it’s the main problem. People will click the wrong thing, whether because of a muscle twitch, not looking carefully, or sneezing at the wrong moment. It is going to happen again. The question is, then what? That’s not being addressed very well.
Meanwhile, in Hawaii, they’ve changed the system so a second person has to do the confirmation, not the employee who clicked the button in the first place. That may help …unless the confirmation message for each choice is exactly the same, and it becomes habit to click “OK” every time.
The unnamed employee who clicked the “wrong button” and then confirmed the click has been “temporarily reassigned” to other duties in the department, which I read as “until this blows over.” Frankly, I sympathize with him, since the real problem is how the system was designed. Hawaii at least worked quickly to get a start on fixing that interface, but it’s merely a start.
Tuesday Update
Here’s the newly “improved” screen for the operator:
Still a Horrible Design
This isn’t a “pull-down menu” as the Hawaiian officials originally described, but notice how pretty important functions are haphazardly mixed with mundane, routine functions. It’s unclear whether the order of any of the items have been changed.
Next, realize the operator has to remember what various acronyms mean. My take: “CDW” means Civil Defense Warning, while “CEM” is Civil Emergency Message. What’s the difference between a “warning” and a “message”? It probably means something to the bureaucrats.
The bottom line: this system needs a complete scrub by a programmer who actually understands user interface design, because this error was downright inevitable.
But It Gets Worse Again!
Now the world knows that the login password for the warning system is “Warningpoint2”. How do we know? Because of all the press hoopla over this, the Hawaiian Emergency Management Agency is doing a lot of interviews, and a reporter noticed that the password was stuck to the user’s monitor, on a Post-It note. You just can’t make this stuff up.
– – –
Bad link? Broken image? Other problem on this page? Use the Help button lower right, and thanks.
This page is an example of my style of “Thought-Provoking Entertainment”. This is True is an email newsletter that uses “weird news” as a vehicle to explore the human condition in an entertaining way. If that sounds good, click here to open a subscribe form.
To really support This is True, you’re invited to sign up for a subscription to the much-expanded “Premium” edition:
Q: Why would I want to pay more than the minimum rate?
A: To support the publication to help it thrive and stay online: this kind of support means less future need for price increases (and smaller increases when they do happen), which enables more people to upgrade. This option was requested by existing Premium subscribers.
I have two thoughts on this:
1. Don’t discipline the employee who sent the alert. They should thank him or her for bringing attention to that flaw.
2. If it is that easy to send a false alert might it also be too easy to accidentally launch an actual attack?
—
Totally different systems, but that system could well have serious flaws too. -rc
My favorite comment on this whole thing was, “Thank God Trump was golfing.”
Ya gotta laugh, or else you’ll tear your hear out while running in circles in a panic like a loon.
The conspiracy theorists (who come in all political flavors) seem to not understand that Occam’s Razor counts here — stupidity is a much easier explanation than malicious intent. There just isn’t any upside in doing this deliberately. Particularly when you (Hawaii Gov’t) then point out exactly how stupid it was.
I think I’m right in saying that this system didn’t even exist a year ago — which means that it was designed quickly, and likely without input from usability, human factors and interface experts. Or they were ignored, to the same effect.
Why Rush, who (contrary to popular belief) is usually fairly rational, is trying to ascribe more to this than there actually is is a bit puzzling. One would think that he’d just be chortling over it happening in a state where the entire political structure is Democrats.
Has Alex Jones blamed UFO’s for it yet?
—
Yeah, the governor “went there” (the malicious intent explanation) almost immediately, then walked it back just as quickly. -rc
Dumb to re-assign the employee. He’s the only employee who now has first hand experience on what can go wrong. He’s the very one you want in that position.
Absolutely! He’s the one person that will *never* make that same mistake again.
I used to work in the nuclear power industry, back when most of the functions were controlled through actual buttons or electronic controls (not computerized). To shut down the reactor, even in emergency situations, took the input of 2 signals, either from the electronics or the operator pressing 2 buttons simultaneously (after first rotating a locking collar to arm the button). Not sure if they still have the manual buttons on the more computerized systems these days, but I know the shutdown logic still requires 2 simultaneous inputs.
This whole thing has been overblown. It was one poor joe’s embarrassing mistake caused by slap-dash software design and bad timing of the alert. Nobody should have to do work that critical at 8:05 a.m. on a Saturday the moment he arrives. Probably didn’t have time for covfefe! 🙂
It’s kinda funny, really. “All comedy is tragedy that happens to someone else.” Laugh, shake your head, be grateful no one got hurt, and move on.
“Nobody should have to do work that critical the moment he arrives.” So no emergencies are allowed to occur at shift change time? What would you say if you arrived at the emergency room with a life-threatening injury at the start of the shift and were told to wait awhile until the personnel had all had their coffee and had settled into the routine?
People are supposed to be trained and ready to perform their jobs as soon as they start the shift. The purpose of this test is to verify everything is ready to go then, and he should have been ready as well. The poor design of the interface made it easy to make this mistake, but the idea that the operator shouldn’t have have to do critical work as soon as he arrived isn’t an excuse.
“So no emergencies are allowed to occur at shift change time?”
Or on Sunday mornings.
Don’t nobody mention Pearl Harbor, now!
I’ve seen too many critical systems built to government (and big business) specifications. The fact that we haven’t gone poof in a puff of glowing smoke has been due more to luck (including, on more than one occasion, the right person at the right time that didn’t believe what he was seeing) than good planning.
For the guy who sent the false alert: Assign him to watch that button and ensure that nobody ever presses it again unless it’s a REAL problem.
Second: Add a prompt: “Please enter your network password to actually send a real missile alert.”
Why do they need an “Are you sure?” checkbox for the test message? That would emphasize the uniqueness of the “Kiss your a$$ goodbye” message….
—
Indeed so, but that’s what I understand from my research on the event. -rc
I want to know if Hawaii or FEMA has been educating people what to do in case of a real alert. On the news, I saw and heard that most parents jumped into their cars and headed for their children’s schools. If it had been a real alert, unless they lived quite close, they would never make it. If they got to the school and the schools are run properly, the children would all be locked into sheltered places, so what did the parent think they could do? The one really bright lady I saw on TV news had somehow gotten a manhole cover off and was putting a child down inside a storm drain.
I also agree with those who said they should thank the guy for illustrating the flaw in their system.
As a long-time Software Support Consultant, I have to wonder how much testing this particular software had before it went live. And whether they thought to set up the test system (if there ever was such) so as to NOT actually send out the messages to everybody in the state. Certainly when I was involved in software testing one of the standing instructions was “think of all the ways the software and the process could be made to go wrong, and try them all out to see what happens”.
I grew up in a ground zero location. We could watch the B-52s (not the band) take off during an alert. A long time ago I decided that if you can’t be far enough away that it doesn’t matter, you should be at ground zero working on your tan. Hiding in your basement just delays the inevitable. Loved your headline.
Someone should tell the woman who put her kids in the storm drain there is a reason workers ventilate before they go in. She is lucky they are alive.
Years ago while working as a fabricator near Seattle we had a standing joke. In the event of a warning we were going to put on our welding helmets and go outside and face Seattle. The thought being we would get a great light show before the shock wave got to us. Mark, you are right, if you live 100 miles away you might have a chance. Just living through the blast is not the only problem. I would rather go in an instant than passing from being irradiated fatally.
For those who worry that missiles could be accidentally launched that easily, relax. The procedure needed to launch them is not only complex, two trained and certified controllers must agree and act together to to authenticate the order and then simultaneously turn the launch keys which are separated so that one person cannot do it alone. There is no connection between launching (DOD) and alerting (FEMA.)
I don’t know how much time they would have if it where an actual attack, not much I’m sure. Still could the confirmation message be followed by one that reads “Sending missile attack warning in 5 seconds (with a countdown) followed by a big ABORT button. As I understand it the operator realized what he had done right after but had no way to stop it.
Unless they at least require that 2 buttons be pushed simultaneously, and place those buttons farther apart than any single person could reach, how can they be sure that a second person is involved? It would be too easy for the same person to say “it’s too urgent to wait until I can get the other guy to come, I’ll just Ok it myself”.
You do have to wonder if there were those who thought they’d settle old scores when the initial Emergency Alert was received … and then, when the retraction was sent, ask themselves, “Now, what am I going to do with all these bodies?”
—
Now that’s funny! (And the answer: “Ah! The storm drain!”) -rc
Having served as an analyst of nuclear attack strategy (ours) many years ago, it is remarkable that they would have something as useless as a missile incoming warning to “Hawaii” with no reference to the specific island. The biggest nukes the NK’s could attempt to launch would have a relatively narrow damage area on any particular island, even if they could hit one and even if the warhead survived reentry and worked.
One of the blogs I follow is Schneier on Security. Bruce Schneier got his start working on computer security, encryption, and software protocols. Now he deals with security systems in general.
One of the things he says about security is that it requires a peculiar mindset. Whenever he reads or hears about a new security system, his first thought is something along the lines of “How can I break it?” He will explore ways to subvert, divert, invert, or just plain pervert the system. Then he’ll point out the flaws and if he’s in a position to do so, try to fix them.
Too many people design systems based on the assumption that the user will do everything right, or at least the way the programmer expects them to.
They often don’t.
Err wow. Looks like the operators will definitely need training on the new stuff.
It boggles my mind that you would redact a PACOM (CDW) with a BMD False Alarm.
Being a child of the 50’s and 60’s when duck and cover was the sage advice for a nuclear attack, I have determined for myself and my loved ones to take potential nuclear attacks in stride and do nothing extraordinary in case of any notice. I hope that if we are attacked we are at ground zero, because we are certainly not prepared to nor want to live in a post-apocalyptic world.
I and my assistant have been managers of a well-respected college radio station. When a student really messes up, the a-hole will tell us “it won’t happen again.” To which the assistant says to me, “until the next time.”
The Hawaii situation: The people in charge will devise a means to make sure it won’t happen again. Until the next time.
I live in Honolulu and experienced last Saturday’s interesting morning threat of nuclear annihilation before I had even finished my first cup of coffee. My daughter and I were the only ones awake when the alert came on our phones. I checked every TV station and every radio station, plus online sites for any confirmation–nothing. We have sirens for both natural disasters (i.e. tsunami) and nuclear attack (i.e. Marshall Kim’s pique) which are tested at 11:45am on the first weekday of every month. (This sometimes causes our many visitors concern as they frolic on the beach just before noon!)
But there were no civil defense sirens to be heard.
I concluded it was a mistake and did not even wake my wife and my other daughter, figuring that doing so without reason would subject me to a more scorching experience than an unlikely nuclear attack.
We went about our planned activities–it was another ludicrously beautiful Hawaiian day–just irritated that it took the powers that be 38 minutes to issue the retraction. The explanation was that the state needed approval from the Federal Emergency Management Agency to issue the retraction, and it took a while for that to happen.
It turns out that is not the case: http://www.civilbeat.org/2018/01/governor-knew-2-minutes-after-missile-alert-that-it-was-false/
Anyway, the whole day was a wake-up call to truly live each day to the fullest and as if it really were your last. Hug those you love, treat those you don’t with kindness. Embrace grace or karma or redemption with each passing hour. Oh, and Live Aloha!
—
That report notes Hawaii Emergency Management Agency administrator Vern Miyagi admitted, “I misunderstood the requirement. I said that we needed authorization. I was wrong. It would’ve saved some time” [to not seek that authorization]. Another interesting tidbit from the report, which is based on hearings on the incident: it was just the 27th test of the system, so it didn’t take very long to get the false warning — which backs my contention that the system was so broken, such a false alarm was pretty much inevitable. -rc
It’s interesting to see this story continue to unfold. Now the news is reporting that, poor UI design notwithstanding, the worker intentionally pressed the real link because he didn’t hear the “exercise, exercise, exercise” at the beginning and end of the statement. The employee himself seems to acknowledge his error was not “pushing the wrong button” due to the interface, but intentionally sending a warning because the announcement used the bizarre statement “This is not a drill, exercise, exercise, exercise,” and he was confused about the nature of the attack. I think some blame belongs to whomever thought that wording was a good idea. I feel bad for the worker who has apparently lost his job and continues to face scrutiny in the press.