Posted: 3 Min ReadExpert Perspectives

It’s Time to Put Tests to the Test

To get results that are meaningful to you, here’s how to evaluate anti-malware tests

Tests are an important part of life. They provide a reality check and help us make important decisions. But for a test to fulfill those purposes, you must understand how it was done and what it measures – and does not measure. Good tests are closely linked to real-world experiences, so the results can then be used to make real-world choices. Moreover, a good tester will say what they are going to do, do what they said, and show their work.  If a test hides important details then you are really just taking the tester’s word for it.  At Symantec, by Broadcom Software, we recommend tests that apply the principles of the Anti-Malware Testing Standards Organization (AMTSO), an international organization dedicated to promoting the accuracy and relevance of anti-malware testing methodologies.

Over the years, Symantec, as part of Broadcom Software, has participated in more tests than any other company, with an outstanding track record. For example, Symantec products were just recognized by AV-TEST for Best Protection for the sixth time in seven years. Symantec products had the highest protection score out of any participants in SE Lab’s 2021 Enterprise Endpoint Protection test. We also scored top marks for protection in 2021 from MITRE and MRG Effitas.  Our products performed better than any of our competitors in these tests, but that doesn’t mean we think the tests are perfect. Security products evolve. Tests, to remain relevant, need to evolve with them.

Symantec products had the highest protection score out of any participants in SE Lab’s 2021 Enterprise Endpoint Protection test.

How can you tell if tests are not keeping up? One telltale sign is uniformly high scores. One recent test had 19 of 20 products scoring over 99%, and 8 of those scored 100%. Results like that make the participants feel good, but don’t give you the comparative information you need to make decisions.

Another issue is the completeness of the tests. Some anti-malware tests don’t check for false positives. Large numbers of false positives are more than an annoyance. They can put an organization at risk by causing managers to tune out the noise. The result has been historic and catastrophic breaches that have caused hundreds of millions of dollars in losses. An FP test accompanying a protection test also serves as a guardrail against gaming of the test. Without an FP test, vendors may set their products to unusable levels of detection that customers would not use.

As we survey the field of anti-malware software tests, we’re finding that some of the most innovative Symantec product capabilities exceed the scope of some tests. For example, many tests do not encompass the capabilities of Symantec’s Adaptive Protection technology.

Adaptive Protection studies the customer environment, then adapts and molds to different users and policies. The more machines and users there are, the more adaptations are made. Too often, however, test results are for the generic case, not the specific implementation. Without taking into account the power of Symantec’s adaptive technology, third party tests will paint an incomplete and misleading picture. So as well as Symantec products have performed in real-world tests, the power of our innovation for your environment will be even better. Whatever your particular environment, it’s specific and unique. Does it really make sense to pay attention to test results that are for a generic environment – an environment other than your own?

Without taking into account the power of Symantec’s adaptive technology, third party tests will paint an incomplete and misleading picture.

Another example is Endpoint Threat Defense for Active Directory (TDAD).  Symantec’s TDAD simulates attacks, continuously probing for misconfigurations, vulnerabilities, and persistence. It also automates attack mitigation and provides managers with real-time breach visibility. If you have AD, you need to protect it against attacks. Does the test cover software designed to protect Active Directory?

In the realm of cyber security software, analysts play important roles in evaluating products as they come to market and helping buyers understand how the products might protect them. So analysts, of all people, should have a deep understanding of tests in order to offer their clients meaningful advice. If you are consulting analysts or analyst reports in your decision-making process, ask about the criteria they apply to product evaluation. Do those criteria reflect the challenges of the real world in which you must protect your organization’s precious data?

Making decisions in the dark is not a good practice, and tests certainly have their place. Make certain that the test that you are looking at provides enough details for you to assess how important the results are to you.  Ask tough questions. Putting tests to the test will get you closer to answers that will help you make critical decisions as you defend your organization from the bad actors.

Symantec Enterprise Blogs
You might also enjoy
Video
2 Min Read

Symantec Endpoint Shines in the 2020 MITRE Engenuity ATT&CK® Evaluations

Finding breaches is good but preventing them is critical

Symantec Enterprise Blogs
You might also enjoy
Video
3 Min Read

How Symantec Adaptive Protection Marks a New Chapter in Security Defense

Security automatically customized to your endpoints with SES Complete

About the Author

Mark Kennedy

Distinguished Engineer, Security Technology and Response

Mark works on threat detection technologies at Symantec, part of Broadcom Software. He served on the Board of Directors of the Anti-Malware Testing Standards Organization; and was also Chairman of the IEEE Industry Connections Security Group's Exec Committee.

Want to comment on this post?

We encourage you to share your thoughts on your favorite social platform.