Stack Overflow 2019 hack was guided by advice from none other than… Stack Overflow

January 27, 2021 TH Author

Developer site Stack Overflow has published details of a breach dating back to May 2019, finding evidence that an intruder in its systems made extensive use of Stack Overflow itself to determine how to make the next move.

At the time, the company reported that an unauthorised person had logged into its development system and escalated their access to the production version of stackoverflow.com. The source code for the site as well as the names, IP addresses and email addresses of 184 users was stolen, but not the databases which contain the content of the site and that of its customers.

Now further details have been reported by Dean Ward, principal developer in the architecture team at Stack Overflow, apparently “after consultation with law enforcement.”

The report describes the timeline of the attack, which started on April 30th with a probe of the Stack Overflow infrastructure. It appears that the source code was a specific target, as one early and unsuccessful move was to pose as a customer to request a copy “for auditing purposes.” According to the report, “This request is rejected because we don’t give out source code and, additionally, the email cannot be verified as coming from one of our customers.”

Despite the poor start, a few days later the attacker successfully logged into the StackOverflow development environment, using a crafted login request that bypassed access controls, and then successfully escalated privileges. They then got access to TeamCity, the JetBrains continuous integration product.

“A misconfiguration with role assignments means the user was immediately granted administrative privileges to the build server,” said Ward.

Although not having secrets in source code seems like a no-brainer, developers sometimes find this hard to avoid

How does TeamCity work? “The attacker is clearly not overly familiar with the product so they spend time looking up Q&A on Stack Overflow on how to use and configure it,” said Ward.

The intruder cloned several repositories hosted on GitHub Enterprise, using access configured for TeamCity. “They continue to browse Stack Overflow for details on building and running .NET applications under IIS as well as running SQL scripts in an Azure environment,” Stack Overflow said.

In what sounds like a serious move, the intruder wrote some SQL to elevate permissions across the entire Stack Exchange network and “after several attempts, they are able to craft a build that executes this as a SQL migration against the production databases housing data for the Stack Exchange Network.”

The community noticed a new user with broad privileges and reported it, at which point the Stack Overflow security team took more drastic steps, taking Team City offline and removing privileges and credentials. Some aspects were missed, though, and the “attacker pull[ed] source code again,” while also viewing questions on how to build .NET applications and (we are told) “how to delete repositories on GitLab.” The infrastructure was further locked down, and the “attacker continue[d] viewing Q&A, this time around SQL and certificates,” in their last reported actions.

Although it appears that damage to the StackOverflow site and the amount of data stolen was small, the company did, it seems, have a lot of source code stolen, although how valuable this is (other than for guiding new avenues of attack) is open to debate.

The incident was revealing though, and not only in proving that bad folk use Stack Overflow too. It showed how the development and build process can be a weak point in IT systems.

Developers may have a high level of access to production systems, and even if they do not, corrupting the build process can be a way of creating backdoors which are then deployed into production.

Twitter API key was in the source code

Stack Overflow went on to describe the changes it made to address shortcomings in its security. “We had secrets sprinkled in source control, in plain text in build systems and available through settings screens in the application,” confessed the team.

It also moved build and source control systems behind the firewall, added metrics and alerting around privilege escalation, and blocked the ability to view account recovery emails within the system.

Although not having secrets in source code seems like a no-brainer, developers sometimes find this hard to avoid. A follow-up thread reveals that a Stack Overflow integration with Twitter was disabled because the Twitter API key was in the source code and the developers have not worked out another way to do it. “We decided the functionality wasn’t critical enough to justify the effort involved,” said Ward.

Future plans include mandating two-factor authentication with a new VPN, building a runtime secret store, and breaking apart build and deployment. Although this goes against the trend for continuous integration, it will, said Stack Overflow, “allow us to have deterministic builds and better manage deployment permissions.”

For every attack like this that is noticed, reported and remediated, there must be others that are not.

Who was the attacker? “We are not able to comment on any other details related to the attacker due to ongoing investigations,” said the company – though it looks like the moment the community spotted the attack was recorded in StackExchange chat, together with the (likely fake) name of the user. ®