An important system on project [REDACTED] was all [REDACTED] up

Who Me? Welcome once again to the horrors of Monday, dear reader. But fear not – The Register is here to cushion the blow of the working week’s resumption with a instalment of Who, Me?, our reader-contributed stories of tech gone awry.

This week meet a reader we’ll Regomize as “Red” – short for [REDACTED]. A decade or so back (we can’t be too specific about the time) Red worked for a US defense contractor on a “special access” project.

What was the project? Red can’t tell us. What did it do? Red can’t tell us. Who was it for? Red can’t tell us. Or he could, but then, you know …

It’s not worth the risk.

Only a handful of individuals apparently had the clearances and accreditations required to work on this hush-hush project, so staff turnover was very limited. It operated in a secure space, with its own walled-off access and control system (which was initially set up by a third-party contractor) as well as a dedicated security administrator. Red’s job – to the extent he can tell us about it at all – was “to provide accredited secure systems to fulfill contract requirements.”

Sounds like “Universal Exports” to us.

Anyway, this whole system apparently ran quite well for quite some time with few changes required – helped, in no small part, by the very low turnover of staff.

Then, the dedicated security administrator left, and unleashed havoc. The secure space became, if anything, too secure.

“One day,” Red tells us, “the system would not allow anybody to enter.” In order for anyone to get into the building, “a manual high security lock had to be opened by the facility security officer.” The facility security officer wasn’t part of the project, but was permitted access. Like, what choice was there?

At this point Red was called in. His first task was to identify what had gone wrong. And his first diagnostic discovery was that the access control software was running on a PC with Windows 95.

We mentioned this was only about a decade or so ago, right? Windows 95 was not exactly the latest and greatest thing. Or secure. Or supported.

Before you start nodding sagely and sheeting home all blame to Microsoft, it turns out Win95 was not the sole culprit. The access control software had littered the drive with temporary files that neither it nor Win95 managed to clean up, with the result that the PC’s drive was both highly fragmented and almost 95 percent full. To add to the fun, the access control software demanded ten percent of the drive be free in order for it to create its temp files – which is madness.

Red tried calling the contractor who had set up the system, only to learn that it was no longer under contract and thus not supported. Plus the super-secret project’s super-secret budget had no spare pennies in it to pay for a new contract.

The fix was therefore up to Red, and Red alone. He did some manual knuckle work to free up some space on the disk so the access control system worked and everyone could get in and do their super-secret jobs. Yay Red.

Then he tried to update the access control software, and found another barrier: the update required an internet connection. Do you think this super-secret computer in a super-secret facility on a super-secret project for a US defense contractor was allowed to connect to the internet?

Yeah, nah.

So Red decided to leave the old system running on Windows 95, but cobbled together a backup and cloning system for it to keep it stable, and wrote some scripts to get rid of temp files on a daily basis. Not exactly what you’d call a permanent solution, but enough to keep it going.

And keep going it did, at the very least until Red left the project – in 2013 – to make it someone else’s problem.

Ah, rugs, and the things we sweep under them. Aren’t they wonderful? If you’ve got a tale of kicking a problem down the road for someone else to deal with, we’d love to hear about it in an email to Who, Me? And we promise, we’ll keep you as anonymous as our old pal [REDACTED]. ®

READ MORE HERE