Be careful of data without context: The case of malware scanning of journaled emails
Recently, we shared details on how effectiveness is measured for Office 365 Exchange Online Protection (EOP) and Advanced Threat Protection (ATP). We also followed up with a comprehensive update on Office 365’s improved ability to stop phishing emails from impacting users. These reports highlighted:
- Enhanced anti-phish capabilities for EOP/ATP.
- Visibility and transparency into our testing methods.
- Performance improvements from the engineering updates.
Today, we’ll cover recent research on a testing methodology—email journaling—which is often used but can lead to misinterpreted results.
What is email journaling?
Email journaling (Figure 1) is when an organization enables recording of emails for retention or archiving. With growing regulatory requirements, organizations increasingly must maintain records of communications between employees performing daily business tasks. Journaling helps organizations respond to legal, regulatory, and organizational compliance requirements by recording inbound and outbound email communications. Exchange Online offers in-depth journaling capabilities. Microsoft provides extensive and up-to-date recommendations on how organizations can manage and configure journaling requirements.
Figure 1. Email journaling mail flow.
The effect of journaling on email security
Sometimes we receive inquiries from customers seeking guidance on whether journaled email can be used to measure the effectiveness of our security solution. Sometimes, third-party vendors use journaled emails to assess effectiveness; however, this can lead to inaccurate results because there is a:
- Misunderstanding of how the email security protection stack is built in Office 365.
- Mischaracterization of a miss versus a catch because of #1.
- Misinterpretation of data to customers based on #1 and #2.
In this scenario, third-party vendor(s) recommended customers create a journaling rule routing emails to the vendor’s testing cluster. The vendor(s) then evaluated the emails with their advanced filters to determine which emails Office 365 ATP “missed.” However, in Office 365, ATP protection—which includes Safe Attachments (file/URL detonation) and Safe Links (time-of-click protection)—comes after the journaling rules. As a result, emails routed to a journaling archive have not yet been scanned by ATP’s Safe Attachments or Safe Links policies (Figure 2). Our stack is designed so that journaling comes after the standard EOP anti-virus scans—but before ATP scans—so known malicious emails are not archived. A potential best practice is to rescan emails released from an archive to help ensure they’re not malicious.
Figure 2. The top graphic shows the entire mail flow and security stack in EOP/ATP, while the bottom graphic is a blowup of the section that shows where the journaling rule takes effect and how it is before our Safe Attachments/URL sandboxing policy, which is part of Office 365 ATP.
Helping ensure our customers’ security
When journaled mail is used to measure effectiveness, it’s important to remember that the emails are not scanned by ATP—since journaling happens before ATP. Figure 3 shows how measuring journaled emails for effectiveness can cause a misinterpreted analysis because most emails thought to have been missed were in fact blocked by ATP and how a large percentage of emails not detected by ATP were false positives.
Figure 3. The emails which were characterized as “misses” never went through the ATP filters. When we ran them through ATP, we found ATP blocked most of the emails and also did not block many emails that were false positives.
There have been situations where customers were advised by third parties to use journaled emails to identify emails missed by Office 365 ATP, which is impossible due to the architecture of Office 365’s mail flow. As with any service, Office 365 ATP also misses some emails. No service is 100 percent secure, but the best services are able to enhance and evolve quickly to address emerging threats. This ability to quickly enhance our services is one of our strengths and is manifest in the rapid evolution of Office 365 ATP into the optimal security service for Office 365.
Flipping the script
Interestingly, customers often provide us with email samples already scanned by a third-party vendor’s advanced filters to determine how Office ATP would perform on the same set of emails. Figure 4 shows the unique catch of ATP versus a third-party vendor in one such inquiry. In that inquiry, Office 365 ATP found 18 times more unique malicious emails than the third-party vendor. Also, with phishing being a predominate form of attack, we saw that the third-party vendor missed several hundred phishing emails.
Figure 4. Office 365 ATP’s unique catch rate is 18 times greater than a third-party vendor, from a recent comparison of data shared with Microsoft by a customer.
How do I know which data to trust?
We don’t look for gaps in third-party services unless a customer asks us to investigate. Our focus is on enhancing our service to help provide maximum security for our customers. We don’t claim to catch everything; however, we are confident that no other service will secure you better in Office 365 than Office 365 ATP. Put us to the test with a trial. We have previously suggested that bifurcating real mail flow will provide a side-by-side analysis on effectiveness. This is the most powerful and informative test.
Begin an Office 365 E5 trial today to experience best-in-class security for your Office 365 environment and contact your rep to help you test us against any competitor. The results will serve as the ultimate validation that Office 365 ATP provides the best security for your Office 365 environment.
READ MORE HERE
