More than 83K certs from nearly 7K DigiCert customers must be swapped out now

As the DigiCert drama continues, we now have a better idea of the size and scope of the problem – with the organization’s infosec boss admitting the SSL/TLS certificate revocation sweep will affect tens of thousands of its customers, some of which have warned that the short notice may have real-world safety implications and disrupt critical services.

A brief refresher on what happened: On July 29, the certificate authority said at least some customers had just 24 hours to replace their previously issued security certificates due to a five-year-old programming flaw in its systems. It’s fairly technical, and you can read all about it here in our earlier article.

But it basically it involves broken domain ownership validation, random numbers, and underscores, all leading to a selection of issued certs being deemed untrustworthy and thus in need of immediate revocation and replacement.

The issue affected “approximately 0.4 percent of the applicable domain validations” that DigiCert had already issued to its customers, according to the CA. But it didn’t put a number on this 0.4 percent.

A certain Mozilla Bugzilla post, however, puts the number in the thousands. “We have identified 83,267 certs impacting 6,807 subscribers,” noted DigiCert’s CISO Jeremy Rowley. “We are planning to begin revoking within the 24-hour time window.”

Some of these customers, however, say the quick turnaround time won’t work. Others, according to Rowley, have filed lawsuits to block the certificate revocation.

The reality is that many large organizations cannot reissue and deploy new certificates everywhere in time

“Unfortunately, many other customers operating critical infrastructure, vital telecommunications networks, cloud services, and healthcare industries are not in a position to be revoked without critical service interruptions,” he said.

“While we have deployed automation with several willing customers, the reality is that many large organizations cannot reissue and deploy new certificates everywhere in time.”

DigiCert will be publishing a full incident report, Rowley added, admitting that this whole snafu has illustrated the real-life difficulties that come with adhering to the 24-hour revocation requirements set by the CA/Browser Forum.

“We are aware of and are participating in the active industry discussion happening about the applicability of revocation timelines given the widespread impact and the relative severity of incidents,” Rowley wrote.

“We also note that browsers have mentioned that delayed revocation might still be acceptable under ‘exceptional circumstances.’ However, given no clear definition of what would constitute an exceptional circumstance, we are seeking root store feedback as soon as possible, as we are standing ready to begin revocations within the timeline.”

An email sent to affected customers this week and shared with The Register reiterated that some DigiCert users with “exceptional circumstances” would be given extra time — but not a whole lot of it.

“Based on discussions with these affected customers, conversations we have had with members of the web PKI community, we are now in a position to delay some revocations, given exceptional circumstances,” the email explained.

Customers needing more time were encouraged to email the company by July 31, no later than 1930 UTC, with a detailed explanation of the circumstances necessitating a delay in the certificate renewal and revocation process. 

But even if a delay is approved, “All certificates affected by this incident, regardless of circumstances, will be revoked no later than Saturday, August 3, 2024, at 1930 UTC,” the notice said. “We will not be able to delay revocation beyond that date and time.”

One affected customer told The Reg they had 96 serial numbers of certificates to replace, and that required the IT team to update certs on almost 200 systems and applications.

“They told us via email on Monday, July 29 that we had until July 30 to swap out all the certificates before they were revoked,” the reader told us. “It took 15 people 20 hours to touch everything. Good thing we noticed the email right away or it would have crushed us.”

Our tipster wanted to remain anonymous to avoid putting their organization at risk. But they also wanted “to emphasize that for some organizations including ours, revoking certificates with such short notice can pose a risk to life and safety systems.”

The reader continued:

This is unlikely to be the last we hear of the 24-hour rule — and the resulting thousands of organizations now scrambling to adhere to it and ensure secure internet communications. Meanwhile, hats off to the IT teams working overtime to manually swap out the flawed certificates with new ones.

Hopefully you get more than a $10 Uber Eats gift card for your long nights ahead. ®

Contact us in confidence, directly here or via here.

READ MORE HERE