A Look at the MITRE Engenuity ATT&CK® Evaluations For Symantec, By Broadcom Software

The results of the MITRE Engenuity ATT&CK® Evaluations have arrived and it’s time to take a close look at them. Evaluating what tests reveal – and what they don’t – takes willingness to look under the covers. As I discussed in a previous blog entry, a test can be valuable if it gives you information that is relevant to your technology decisions and you’re willing to trust your own judgment.

The Results

Released March 31, 2022, the MITRE Engenuity ATT&CK® Evaluations covered 30 vendors and emulated the Wizard Spider and Sandworm threat groups. Wizard Spider is a financially motivated criminal group that has been conducting ransomware campaigns since August 2018 against a variety of organizations, ranging from major corporations to hospitals.

Sandworm is a destructive Russian threat group that is known for carrying out notable attacks such as the 2015 and 2016 targeting of Ukrainian electrical companies and 2017’s NotPetya attacks. According to MITRE, these two threat actors were chosen based on their complexity, relevancy to the market, and how well MITRE Engenuity’s staff can fittingly emulate the adversary.

MITRE Engenuity evaluated our product, Symantec Endpoint Security Complete (SESC), covering both detection and protection. For the detection results, Symantec, as part of Broadcom Software, provided Analytic coverage on 100% of the steps tested by MITRE – 10 steps emulating Wizard Spider and 9 steps emulating Sandworm. These results show that we flagged suspicious activity at each step in the attack chain.

As the name implies, attack chains are composed of individual links, or steps. Test cases string together a number of links to make a sub-chain of the attack. When evaluating whether or not protection is expected or proper for this sub-chain, there are a couple of considerations. The chain must be evaluated in isolation. One cannot inject “special knowledge”, such as the fact that a certain actor is “known to be bad”. In this regard, the sub-chain itself must include malicious activity explicitly. Saying “it is going to do X” is not proper if X is not performed in the test case. This is particularly relevant to some of the test cases in the ATT&CK® Evaluation.

The protection test is worth close scrutiny. We blocked the beginning and end of the attack chain, but not test 4. That’s because that test covered common activities that were not in themselves malicious. To block them, in our view, would be to return false positives. Although our software could be tuned to block test 4, that would not be helpful to cyber security managers because the “noise” of false positives can inundate staff, which can leave an organization open to devastating attacks. If, however, an organization considered those activities in themselves suspect, our Adaptive Protection feature could be adjusted to block them without flooding managers with false positives.

The Details

Test 4:

1. cmd.exe executes vssadmin.exe to create a copy of C:
2. cmd.exe executes reg.exe to save HKLM\SYSTEM

Others may differ, but we think our interpretation of test 4 is correct. In fairness, this is only the second year MITRE Engenuity has run this test and it is not yet mature. As noted in my previous blog entry, the MITRE Evaluations lack a false positive test, which we believe is needed. And as noted in this blog entry, Adaptive Protection is an important feature of ours for which there is as yet no effective test. 

Any products that blindly block these activities out-of-the-box without any training would generate false positives in the real world. Symantec’s Adaptive Protection normally blocks these activities using behavioral machine learning if they do not typically occur within an organization. If our customers’ endpoints never perform these behaviors in real life, they would have been blocked by Adaptive Protection. The learning period used by Adaptive Protection is often skipped during lab testing, leading to behaviors getting logged instead of blocked, which you can see in the MITRE Evaluation.    

Let’s look more closely at the use of vssadmin.exe in test 4. A product could choose to block this tool’s execution, but without telemetry regarding legitimate usage to back that the risk of Business Impact from blocking is unknown.  Since the ATT&CK®Evaluations environment does not conduct “normal business” activity - not even a False Positive test - blocking this completely legitimate and documented usage of the tool is speculative at best.

For the sake of perspective, the MITRE Engenuity ATT&CK® Evaluation is only one of many. We at Symantec, by Broadcom Software, have a long history of positive protection test results, including outstanding results in the 2020 ATT&CK® Evaluations. In addition, we have earned:

• The top grade in SE Labs (AAA) since Q3 2012.
• The top grade in MRG Effitas (Certified Level 1) in 13 of the last 15 quarters.
• Best Protection from AV-TEST for 6 of the last 7 years.

You’ll hear a lot about the new ATT&CK® Evaluations results in the coming days and weeks. When you look at these and other test results, keep in mind a test’s purpose, its limits, and what it is telling you. Look in the results for information you need to make an informed decision about your own environment. Ultimately, responsibility for protecting your organization’s data rests with you and working with a trusted partner. Contact us here to see how we can work with you to secure your enterprise.