There's been a bit of a fuss lately about a
New York Times article
that said that Baltimore city computers were hacked in part by an exploit
known as EternalBlue that was stolen from the NSA a couple of years ago.
I don't know if the attackers targeted the vulnerability used by
EternalBlue, nor if they used the actual purloined code. What's more
interesting to me is that Microsoft
for the vulnerablity in March 2017, but many city government
systems remained unpatched and hence vulnerable. Why? What did Baltimore
(and other organizations, especially governments) do wrong?
Why were these systems unpatched and hence vulnerable for so long?
(A semantic note: a
vulnerablity is the actual flaw in some software package.
An exploit is the code that takes advantage of it.
EternalBlue is NSA's exploit for the
Don't get me wrong; as I've written before,
patching is hard and risky.
I also noted that sometimes, it's
important to take the risk.
(And yes, in that latter blog post I was writing about EternalBlue.)
But this incident shows a more serious failing: EternalBlue and
other exploits targeting that vulnerability have been
a threat for more than two years. Baltimore's systems have remained
unpatched for that long.
The root of the problem is attitudinal: in many places,
computers are treated as
capital equipment with a fairly long lifespan, and as devices that need
operation but not maintenance. These attitudes may date back to the
1950s, when the first was fairly true and the cost of maintenance was
hidden in the operational cost. Neither is true today. Computers are
consumables that require regular, skilled care. Skipping
this care is like not changing the oil in your car: you can get away with
it for a little while, but at some point you're in trouble. In fact,
and as I explain below,
it's worse than dirty engine
oil: not only are you at risk for a security
incident, you end up in a maintenance trap.
To paraphrase George R.R. Martin's famous line “valar morghulis”,
all code must die. In fact, software is rarely healthy even to start
with; vendors constantly issue patches for their products. Eventually,
though, the patches stop: there's a new version of the product, and vendors
have no interest in continuing to support ancient versions. Not only
would they rather sell you something new, there's no viable economic model
to pay for continued patch development for older versions. The cost of
the first few years of patching is, of course, built in to the initial
price of the software.
Ultimately, of course, a software package is succeeded by a newer
version. Switching versions is in some sense the ultimate patch,
and just as patching can be hard, upgrading can be very hard.
Worse yet, patching and upgrading are expensive. System administrators
and application programmers need to test and perhaps modify their
code for compatiblity with the new version. Extra hardware may be needed,
partially as test machines but also because at some point, newer vendor
packages (especially operating systems) simply won't run on older computers.
If money is tight, it becomes very tempting to postpone patches and
upgrades. After all, the reasoning seems to go, if things are working
now, why bother changing?
Government agencies are particularly vulnerable here. Year-to-year
budgets aren't that predictable; they're at the mercy of political
winds, and tax increases are never popular.
(Partisan anti-tax rhetoric is another factor, of course.)
If there's a funding crunch, deferring software maintenance is an easy
thing to cut, especially since it's hard to lay off civil servants
and city governments can't easily sell off agencies.
(Privatization? That's a political question I won't go into, and ultimately
it doesn't matter: whatever the irreducible core of governmental
functions, that core will be susceptible to the same dilemma.)
So: software maintenance is deferred. What are the consequences?
If the problem is a delayed security patch, your site is vulnerable until
the patch is installed. But if you miss a version upgrade, you're in
big trouble. For one thing, security patch support tends to end soon
thereafter. For another, upgrading to the next version is much harder:
there are generally tools to help you migrate to the next version, but
tools that reliably upgrade two versions are much, much harder to build,
and may not even be available.
The net result is a system that is insecure, unsupported, and more
expensive to maintain and upgrade than one that had been patched and upgraded
properly all along.
Ultimately, this boils down to money: where will maintenance money come from?
The way to solve this problem is to accept, in a tangible form, that a
computer is first, a consumable item, and second, one that requires ongoing
expenditures. Think of it as a leased automobile: while you have it, you
have to insure it,
refuel it, and change the oil and tires; when the lease is up,
you have to replace it—or do without a car as a means of transportation.
In other words, a leased car costs you something to own and operate, and
has a relatively short lifespan.
It's the same with computers. An enterprise computer should be regarded
as having a lifespan of about four years, and an enterprise's budget
should include money for the expense of operating and maintaining each
and every computer it owns.
Furthermore, budgets should include money for operating system and major
application upgrades. They will happen, and doing these upgrades
will cost money—but these upgrades are utterly necessary.
(No, I'm not saying that a corporation should instantly switch to the
next release of WhateverOS as soon as it comes out—I learned “never
install .0 of anything” back in 1970—but not all that long after
WhateverOS X.0 comes out, release X-1 will drop out of support.)
In a well-run corporation, the CIO would put advice like that into
practice. (In a corporation with a well-run IT department, they already
know this and do it.) It would be an interesting exercise to try to
mandate it for government agencies, either by law or by executive order.
I'd be interested in suggestions on how to do it.