Former NSA cyberspy’s not-so-secret hobby: Hacking Christmas lights
Video In 2018, Rob Joyce, then Donald Trump’s White House Cybersecurity Coordinator, gave a surprise talk at the legendary hacking conference Shmoocon about his hobby.
As the former head of the NSA’s Tailored Access Operations squad – the people who crack systems and gather intelligence for the US government – Joyce was also the friendly public face of the agency. The agency didn’t come out of the Edward Snowden affair with a great reputation when the ex-NSA contractor-turned-whistleblower made public the existence of the NSA data collection programs back in June 2013. Many in the security industry were peeved at the agency’s disregard for privacy and the accepted norms under which people assumed it operated.
Joyce was part of a campaign to make the NSA acceptable again, and he was doing a good job. We covered his talk at the first Enigma security conference, and it was unusually frank – most talks by agency personnel are about as exciting as watching paint dry.
But the Shmoocon talk (see below) was a personal matter. It turns out Joyce is a big fan of Christmas, and of hacking the seasonal strings of lights that hang off so many American houses during the holiday season. As you can see from the video below, he’s serious about it and applies all the rigor normally used to break into adversaries’ networks to put on a show.
Next year, at the last Shmoocon, Joyce will reprise his talk and offer updates on how to turn your house into a light show that will either delight, or irritate, the neighbors. He spoke to The Register ahead of time to explain the hobby.
The Register: The first and most obvious question: why? What got you into this?
Joyce: We had a family tradition where we drove around looking at Christmas lights on Christmas Eve with the kids. And we passed a house only a few miles from us that had computer-controlled LEDs synchronized to music and a little radio station you tuned to. And it just captivated me.
It was not elegant, right? It was over the top and gaudy, and just really made me happy. I said “I think I could do that,” meaning I have the technical chops to achieve it. And [Joyce’s wife] said, “yes you can,” and I took that as license to mean, “yes, you can do it.” And so when boxes started arriving in the mail in February and March, she’s like, “what the hell is this?”
The Register: In terms of work time, how long does a setup like this take?
Joyce: The startup was enormous if you’ve seen the images. I could cover the house and put things in the yard, and there are a bunch of different props that are created. If I knew how much work it was I would not have started.
But now the incremental work is not so bad. It takes me about three and a half days to get things set up. I hope for a weather window, and the first day I start is somewhere right before Thanksgiving because the first thing I do is the very top of the roof, up on the outside of the house. The second day is the next layer of things on the house, and then after that, if we have bad weather, I can do things on the ground and around the yard without much concern – but I don’t want to be on the roof in bad weather and wind.
The Register: A senior person in the NSA ordering huge amounts of electronic equipment from China didn’t set off any red flags?
Joyce: None of the compute comes from China, just the LED strings themselves. I would applaud somebody if they could supply chain that.
I do take a little more care in the control system itself. It’s not connected to the internet and is a standalone network – because I do have friends who have interesting hobbies and would love to change my display and make it say some interesting things.
The Register: You’ve said that the things that impress many people aren’t the lights themselves but the radio station that synchronizes with them. How’s that set up?
Joyce: It’s probably on the hairy edge of what the FCC would approve, but I make sure it’s not interfering with anything in the neighborhood. It’s probably the least technical part of the whole creation. You plug an audio signal in and it just spits it out as a radio channel. I did custom make the antenna to make sure that there’s good reception.
The Register: What about the software, do you code it yourself?
Joyce: I don’t write the software. I use an open source product called xLights that will let you map the songs to your display. And there’s a group of people out there who write that software and improve it constantly. I’m very grateful to them and donate to their project every year to help keep them going. But much like me, they do it as a labor of love.
Then there’s some software that runs on Raspberry Pis that controls the flat panel displays that are hung on the outside of the house, and also pipes all the pixel commands in real time out to all the lights across the whole show. That also is open source and part of this hobbyist community.
The Register: Is the Christmas lights hobbyist community that large?
Joyce: Well, every year xLights put out a song and they put out the light sequence to it, and then all these people map them to their display and record their house. Then they cut together a video of all of these different houses doing the same song. And there’s thousands of people that do that. It got so big. I did it for a couple of years, but I stopped doing it just because it’s so big.
The Register: Are you still using the same controller board?
Joyce: No, they’re out in the cold, and they get a lot of thermal cycling. That makes them a little flaky. The technology has advanced now. Now it’s all surface mount technology, and they include better debugging to help you figure out where your issues are. And there’s a Raspberry Pi as the main controller.
The Register: And for connections – wired or wireless?
Joyce: I still do Ethernet. You know, wireless introduces delays and I worry about interference and the signal. Also I still don’t want the mischievous friends coming over and messing with my wireless. They can roll up with a cable of Ethernet and jack into the local network there, but the security cameras outside would spot them.
The Register: What do the neighbors think of all this?
Joyce: I’m trying to keep my neighbors reasonably happy. They all enjoy the light show to an extent, but these days they can’t get in their driveway because the traffic is backed up. You know, that makes for unhappy neighbors. I had one family and was getting the vibes that they weren’t that crazy about it, but they’ve had children and the grandchildren love it, and I think it reminded them of the joy it brings to other visitors.
When there are low lying clouds, you can be a mile away from the house and you’ll see the clouds change color, so it’s always going to be nice thinking, “Yeah, I did that.”
The Register: What advice would you give to people that want to get into this?
Joyce: Have an understanding partner.
Seriously, go to the xLights website. There’s a forum there that will link you to the YouTube community. There are great tutorials on YouTube, and there’s a bunch of very active Facebook groups, and on the xLights site you’ll find something people refer to as the xLights Zoom Room. It’s a group of people who volunteer their time and they sit on Zoom calls and help people work through the technical issues they have.
The Register: Can you give us a rough idea of what the setup on something like this would cost?
Joyce: Oh no, I’m from NSA. I know how to keep a secret.
Joyce also has a Twitter account devoted to his hobby. For security (and traffic) reasons, we won’t be publishing his physical address. ®
It’s spreading to Europe …
American-style Christmas lights aren’t as popular in Europe, but they’re catching on, especially as hackers get in on the game.
The Register spoke to a British security operations center manager and volunteer admin for xLights, who developed a passion for fiddling with lighting after seeing the 2018 talk, and there are about 1,300 enthusiasts in the scene in Europe. This year’s spectacle looks superb, but there are issues that Joyce doesn’t have to deal with.
Regulation is a problem, Paul Glavin explained. As of next year, US manufacturers might not be able to sell the control systems for such lighting displays in the UK. So a grassroots startup Buildalightshow is building its own control boards to brighten up the Solstice night.
“We now have a UK-based board – it’s called a Baldrik board. The network that they operate on is called the turnip network. It’s just that kind of humor behind it [Ed: This is a Blackadder reference, as they have a cunning plan]. So it just gives you a bit of a giggle.”
And while Joyce can broadcast Christmas carols via radio, that’s not really allowed in the UK, so Glavin has speakers on the lawn instead. As a result, he only runs the lights for an hour each day to avoid being impolite to the neighbors and to cut down on traffic, but the project raises money for local school meals.
“The main reason I do it is the love others have for it. I’ve just come in from the 17:30 showing, which had kids dancing and singing along to the songs,” he said on Monday. Given last year’s soundtrack, that’s understandable.
“I had parents telling me this is their x year seeing the lights and this was the only thing their daughter wanted to see this Christmas, other families have mentioned this year that it’s their family tradition; so no matter how hard it gets to put the show on, how much I hate being up a ladder, how much I curse whichever storm has ripped through the garden and damaged yet more props, or more LEDs have failed, I have to keep it going for the local community.”
READ MORE HERE