Symantec Adaptive Protection in 3rd Party Evaluations

Introduction

Symantec Endpoint Security Complete includes Adaptive Protection (AP); a new groundbreaking technology that provides protection against living off the land (LotL) cyber attacks, which leverage legitimate software and functions available in the system to perform malicious actions.

LotL cyber attack operators explore systems for tools, such as operating system components or installed software, then leverage what they find to conduct the attack, often without leaving any artifacts or traces (classified as fileless). Previously, it was very challenging to provide protection against LotL attacks. This is because most computers are general purpose machines, equipped with a rich operating system, such as Microsoft Windows, and filled with powerful applications, tools/utilities, and techniques to run a wide variety of tasks in a wide variety of ways.  It is important to note that all of these are trusted and legitimate mechanisms for performing necessary tasks on these machines.  Most, if not all, users and IT administrators take advantage of some of these applications, tools/utilities, and techniques, in some ways.  However, that leaves all of them available for cyber attack operators to use, in all ways.

How Adaptive Protection Works

Adaptive Protection monitors an organization's environment for a period of time to determine which applications, tools/utilities, and techniques are actually used, and in what ways (behaviors), within that organization.  This usage will vary from “never used” to “occasional” to “often”.  AP makes it easy for the administrator to see and visualize the usage analysis and policy recommendations, then block the application/tool behaviors not necessary for business operations in their environment.  AP removes these unnecessary behaviors from the pallet available for attackers to utilize.  This has many benefits, and can be done with a high level of precision so there is no impact to business operations and productivity.

Adaptive Protection tailors the machines throughout the organization to permit only what is necessary to perform their desired business activities, reducing the attack surface greatly, and making a unique exact-fit security profile.  Moreover, this exact-fit security profile is further adapted at the organizational group level, meaning that even within a company, the attack surfaces vary from group to group.  Attackers, with no outside knowledge of a given target’s profile, cannot make use of a generalized scheme of attack.  Additionally, since each group within the organization can have a different attack surface, lateral movement becomes increasingly difficult; what works on one machine may not work on another.

For more details on this feature and its benefits please see “How Symantec Adaptive Protection Marks a New Chapter in Security Defense”

One Size Fits One – A New Era for Endpoint Security

Most endpoint security solutions use a “one size fits all” approach to their protection.  While this is simple to manage, it does lead to the situation where if you can beat one machine’s protection, you can beat them all.  Attackers can work to craft their attack against a given product within their own network, confident that the same attack will work on all machines running that same security product.  Adaptive Protection breaks that paradigm – one size fits one is the new era of endpoint security.

AP allows for every machine within a group to run a unique protection profile that continuously adapts to changes within the customer’s environment as well as the threat landscape.  Targeted and general purpose attacks will likely have great difficulty operating, because one or more steps in their attack chains can be blocked by AP. 

The 3rd Party Testing Problem

Adaptive Protection is a great capability for our customers; however, it causes great difficulty for 3rd Party testers. Like attackers, 3rd Party testers assume a uniform configuration for a security product prior to testing it.  This is presumed to be an accurate representation of a customer’s environment.  However, testing occurs in simulated environments, which do not perform real work.  As such, Adaptive Protection does not have the opportunity to observe, analyze, and adapt; thus, a test environment is not representative of what a customer’s environment will look like.  This means that test results derived from a non-representable environment, are not representative of what a real customer of Symantec Endpoint Security with Adaptive Protection would see.  Moreover, since every real environment will be different, there no longer is any way to extrapolate results from one test environment to any given customer’s environment.  Simply said, there no longer is any way to simulate the “real world” in a “3rd Party real world test”.  Even the concept of “typical environment” no longer exists.

Adaptive Protection effectively breaks all generalized real world tests.  See our Blog “It’s Time to Put Tests to the Test”.  In our data analysis, for any given environment, 66-75% of all AP behaviors are never observed and can be set to “Deny” with no impact to normal business operations.  However, the specific set of 66-75% behaviors vary greatly from organization to organization, and even vary from group to group within an organization.  This means there is no “typical” environment.  Results from any one environment almost certainly do not represent the results from another.  Without the ability to extrapolate results, the value of the results from any 3rd Party test is greatly diminished.  It is simply not possible to evaluate how well a product will function in a given environment without actually testing it in that environment.  Given that Adaptive Protection includes nearly 300 application/tool behaviors that are monitored and configurable (which will grow over time), it’s computationally and practically impossible to simulate all the possible configurations.

How to Solve the Problem of Testing Efficacy with Adaptive Protection

Given that 3rd Party testing will remain, and given that every configuration is unique, what can be done?  Clearly, some “simulated” configuration is necessary.  But, how do you do that?  It is important that the method utilized to author the AP policy, must be simple to communicate, and be fair.

The best proposal to date to partially address this problem is to utilize another component of a competent 3rd Party test:  The False Positive (FP) test.  In this proposal the FP test takes on the role of “normal business operations”.  While not really accurate, it does put the configuration control into the hands of the tester.  Here’s how it would work:

1. The FP test is run, with Adaptive Protection in "monitor mode".  Any detection events from this test for Adaptive Protection Behaviors are considered as “normal business operations”.
2. All AP Behaviors without detection events (those that did not fire in the FP test) are then switched into “Deny” (blocking) mode.
3. (Optional) The FP test is run again, and there should be no difference in the blocking (i.e. no blocks should occur).  Steps 1 through 3 should be done quickly, so as to prevent normal security company operations from affecting the results.
4. The Protection test is run.
5. Any blocks from Adaptive Protection are counted as blocks.

As stated, this is an imperfect proposal, because there is nothing that will actually yield real results short of using an actual configuration from a company – and those results would still only apply to that company.  However, this does provide “a way” to include this valuable feature.

Here are the pros:

• Somewhat simulates the way Adaptive works in the real world.
• Allows for the feature to be tested without a “canned” configuration (canned configurations are not able to simulate anyone in the real world).
• Gives the tester control over the configuration in their test.

Here are the cons:

• The FP test could be overly broad or overly narrow, resulting in a strange configuration.
• Violates the precept that TP/FP testing should be under the same configuration.  This is mitigated by full disclosure in the methodology.

What Does It All Mean Going Forward?

As stated earlier, this does not solve the problem with real world tests no longer being real world.  Simulating the real world, while difficult to begin with, is now simply no longer possible.  You can simulate a real world environment, but not the real world.  This compromise is merely that: a compromise.  It does not make the results any more accurate for your environment, but it does create a reasonable guess to a theoretical environment, one derived from observed and simulated real world actions.  It is also crucial that the configuration used be posted along with the report.  This transparency should already be present, but it is more important than ever, if the results have any hope of being consumable.

For consumers of 3rd Party tests, it involves a little more work.  They must compare the tested configuration to their own, and assess how closely it matches.  The closer the match, the more relevant the results.