Stack Overflow: Here’s what happened when we were hacked back in 2019

January 28, 2021 TH Author

Stack Overflow, a popular site amongst developers, has revealed more about a week-long breach that it disclosed in May 2019.

Stack Overflow said at the time the attackers accessed user account data, and now the company says that after consulting with law enforcement, it can reveal more about what happened and how a newly registered user came to have moderator- and developer-level access.

Last year, Stack Overflow said it had identified “privileged web requests that the attacker made that could have returned IP address, names, or emails for a very small number of Stack Exchange users.”

According to the brand’s latest update, the hacker accessed and stole source code but it says the breach only affected 184 users.

“A user that nobody recognised had gained moderator and developer level access across all of the sites in the Stack Exchange Network. Our immediate response was to revoke privileges and to suspend this account and then set in motion a process to identify and audit the actions that led to the event,” said Stack Overflow’s Dean Ward.

Ward says the the escalation of privilege was “just the tip of the iceberg” and the company soon discovered a lot more including the exfiltration of source code. Additionally, the breach exposed 184 users’ email, real name, IP addresses details across the Stack Exchange Network.

“Thankfully, none of the databases—neither public (read: Stack Exchange content) nor private (Teams, Talent, or Enterprise)—were exfiltrated. Additionally, there has been no evidence of any direct access to our internal network infrastructure, and at no time did the attacker ever have access to data in Teams, Talent, or Enterprise products.”

Ward provides an account of the attackers activities from April 30 — the date the attacker started probing its build and source code control systems — to May 22, the date Stack Overflow notified affected users of the data breach. The account describes compromise techniques and technical exploits carried out over several weeks in May.

On May 1, someone posing as one of Stack Overflow’s enterprise customers submitted a request for a copy of source code for an audit. The company rejected that request because it doesn’t hand out its source code.

The next day the attacker used a spoofed email address of a customer to raise a support ticket with Stack Overflow. This attack avenue was discovered after Stack Overflow sent an automated reply to the customer whose email was spoofed.

By Friday May 3, the attacker started poking around Stack Overflow’s public facing infrastructure and by Sunday the attacker was able to successfully log in to the development tier.

“Our dev tier was configured to allow impersonation of all users for testing purposes, and the attacker eventually finds a URL that allows them to elevate their privilege level to that of a Community Manager (CM). This level of access is a superset of the access available to site moderators,” explained Ward.

After that, the attacker user the site’s account recovery feature to recover access to a developer’s account. The attacker couldn’t intercept the recovery email, but could use a feature on the dev tier that shows the email content to community managers. The attacker used this feature to get the link to reset credentials.

“This is used and the attacker gains developer-level privileges in the dev environment. Here they are also able to access “site settings”—a central repository of settings (feature flags) that configure a lot of functionality within the site,” writes Ward.

A positive note was that Stack Overflow’s login to its GitHub Enterprise instance was protected by two-factor authentication. But by Thursday May 9, the attacker pulled more repositories from Stack Overflow and then tried to use a virtual machine from Microsoft Azure to connect to the site’s VPN using previously acquired credentials.

Then the attacker starts using Stack Overflow’s own knowledge base to learn how to build .NET applications and run SQL database scripts in Azure that would later be used to attack Stack Overflow. Eventually the attacker creates a method for using SQL to elevate permissions across the Stack Exchange Network.

“After several attempts, they are able to craft a build that executes this as a SQL migration against the production databases housing data for the Stack Exchange Network,” notes Ward.

“Shortly after execution of the SQL, we were notified of the odd activity by the community and our incident response team started investigating.”

Stack Exchange engineers didn’t know the extent of the attack but further investigation revealed a TeamCity account was compromised and was subsequently disabled. Eventually it took TeamCity offline entirely.

“Once we discovered that the escalation path involved dev and the use of site settings to acquire credentials, we committed code to remove those paths—notably, the tool used to view an account recovery email and the site settings used to compromise the TeamCity service account,” notes Ward.

StackOverflow’s analysis also includes a set of recommendations for others:

Log all your inbound traffic. “You can’t investigate what you don’t log.”
Use 2FA. “That remaining system that still uses legacy authentication can be your biggest vulnerability.”
Guard secrets better. “Educate engineers that ‘secrets aren’t just passwords.’ Protect SSH keys and database connection strings too. When in doubt, protect it.”
Validate customer requests. “The more unusual a request from a customer, the more important it is to verify whether or not the request is legitimate.”
Take security reports seriously.