The Linux box that runs the exec carpark gate is down! A chance for PostgreSQL Man to show his quality

On Call This week’s episode of On Call, as ever, comes with a warning: Be careful moving that beige box, for you may not realise what it does.

Register reader “Jim” was the recipient of today’s super-urgent callout, which occurred during his final week of paid employment ahead of a well-earned retirement.

Jim was a 20-year veteran at his company, having worked in the development team that ran the multinational firm’s overseas factories. Describing himself as the resident PostgreSQL evangelist, he’d been given “the talk” by his boss and, like so many in the IT world, found his department was to be dissolved and the work sent abroad.

Luckily, he was near enough to retirement to opt for a life not spent toiling under The Man and used his remaining six months of employment to wind things down. He had also spent the last few years telecommuting, thus avoiding being shoved into a maze of cubes with the rest of the old guard.

“I wasn’t the only one telecommuting unofficially,” he protested before admitting, “even though I might have been the most flagrant.”

And so it was that on his very last Monday, an 8am call woke him at home. On the line was an impossibly young IT employee seeking help with the server responsible for the security gate to the underground carpark at the Silicon Valley HQ.

“This kid tells me, the garage security gate doesn’t work, corporate mucky-muck is pissed he can’t park his Mercedes in his reserved space, and demanded IT mobilize all resources to fix this RIGHT NOW.”

We suspect that Jim meant the bigwig was jolly annoyed (“pissed off”) rather than on the vodkas, but then again, our grasp of American can sometimes let us down.

Unused to being roused at such an early hour, and before he had a chance to hit the morning coffee, Jim asked for as many details as his colleague could muster: IP addresses, passwords, and so on. His knowledge of PostgreSQL and Linux had made him the man of the hour.

Donning the garb of the telecommuter (hoodie and sweatpants), Jim trudged to his home office to fire up his PC and connect to the corporate VPN.

And there it was – a slightly whiffy version of Linux squatting on the network. He could SSH to it with root. It ran the security gate and was physically located in a security office. Somebody, it transpired, had unplugged it to rearrange the plugs and the elderly thing had declined to reboot properly.

So now the gate wasn’t working and the bigwigs’ Mercedes, BMW, and other execu-barges were at risk of either bird guano or an accidental keying from a soon-to-be-outsourced employee.

“It was an ancient version of Linux,” said Jim, “with a really old version of Postgres and a bunch of weird Japanese software that ran the gate controller. Best I could tell on first look, the gate software was shutting down because it couldn’t connect to PostgreSQL, and PostgreSQL was exiting because _____ no logfiles, all had been set to /dev/null.

“Uh oh.”

It took two days to fix. It turned out that PostgreSQL logging was disabled so Jim had to sort that out. Then he found that invalid changes had been made to the configuration files. Since the box had never been rebooted those hadn’t been a problem… until now. And so it went on.

Eventually, with mere days remaining until a well-earned farewell beer with his fellow veterans was scheduled, the resurrected server was ready for user acceptance testing.

“OK,” he admitted, “kid ran out to the parking lot and waves his security card at the gate kiosk, and IT OPENS! Win win!”

“Did I mention, they told me the last guy who ran this box was laid off six months earlier?”

Sadly, Jim never saw the line of bossmobiles filing into the carpark. Instead he pretty much took his final Friday off and headed for a local outdoor craft beer garden and farewell session with his colleagues.

“Which was indeed epic.”

Ever found yourself taking a call with mere days of your notice remaining? Or did you depart knowing you’d left an accidental timebomb ticking behind you? Share the time you were called out for somebody else’s dirty deed with an email to On Call. ®

READ MORE HERE