Ooh, an update. Let’s install it. What could possibly go wro-
Who, Me? Welcome to another Who, Me? confession from the Register readership, and a reminder of the unexpected side effects of software updates.
Today’s story, from “Ralph” (not his name), takes us back nearly two decades to when he was responsible for a selection of servers, this being in the days before virtual machines were ubiquitous.
He had a selection of kit on his books. A SQL Server, an AS/400 doing accounting duty, and a Hewlett Packard server running Novell Netware and responsible for file management and email.
He also took care of the fleet of desktop machines, “having to root out those who spent a little too much time on some more of the shady sites at the time.”
“I was,” he said modestly, “god of my domain.”
The days passed uneventfully. Ralph dutifully ensured live virus scans were run against the servers and desktop machines in order to keep the network squeaky clean. Updates were rolled out. His halo as god of all things IT burned brightly.
Things inevitably took a turn. A seemingly innocuous Novell update turned up and was installed. After hours, of course. It went off without a hitch and the users were oblivious (“as they should be,” said Ralph).
A few days passed. The aforementioned virus scan happened. And all hell broke loose.
The helpdesk was suddenly inundated. We’ve all experienced the crippling of desktop machines when IT, in its infinite wisdom, sees fit to run a virus scan. But this was more serious; nobody could access the Novell box. The box that handled email and files.
Ralph hurried to the console to find no running tasks, no file serving, and lots and lots of errors. How could this be? He checked the obvious and, sure enough, there was no free space on the disk and Novell’s finest was not happy. But how? There had been no warnings, no hints of the impending catastrophe. What had happened?
Naturally, the finger of blame swung to point at Novell. After all, an update had been installed a few days previously and now the world had seemingly ended. However, after more than a day of research and lengthy periods on hold to the vendor’s support lines, Ralph was no closer to identifying the source of the borkage.
Right up until he came across a forum thread from another user having a similar problem, except this was regarding the virus scan software. It transpired that it had also had an update. One which did not play well with Novell. Not at all.
The disagreement between the tools resulted in the file server reporting file sizes as being 10 times normal. The drive was effectively full, even though it really wasn’t.
“The only way to fix this particular issue,” said Ralph, “was a complete rebuild and restore from backups, or get the machine running so the patch that was downloaded would correctly fix the file size issue.”
Like all good admins, he had a spare hard drive space squirrelled away, unallocated. Just in case.
Sacrificing his stash, he rebooted, applied the patch to fix the patch (very Microsoft, these days) and… all the file sizes were corrected and the files themselves accessible once more.
“Mail was once again flowing, and files served,” he said.
“I would love to say the users remained oblivious, but alas, they were all quite aware of the day-and-a-half outage caused by some software that did not play in the sandbox with others.”
While patching has spread to every part of modern life (no, we don’t know why our microwave demanded a software update last weekend either), quality has not always kept pace. Tell us about the time your faith in the in the ability of vendors to deliver patches without a blast radius received its first knock with an email to Who, Me? ®
READ MORE HERE