Utilizing Anomaly Detection for Application Allowlisting

The ability to identify anomalies within a computing environment is critical. Anomalies—events outside the norm—sometimes indicate security incidents, and usually prompt investigation. Whether they ultimately represent a security threat or not, anomalies are a leading indicator of something gone awry, which is why anomaly detection is a powerful tactic.

Many organizations are interested in applying the idea of anomaly detection as a key component of application allowlisting . But it’s difficult to do well, and its success depends in large part on the data that informs it. In this article we’ll explain the basics around application allowlisting , the challenges around implementing it, and show how security teams can use the Uptycs platform to successfully detect unwanted applications executing on endpoints.

What is application allowlisting?

The idea behind application allowlisting is simple: A security team creates a list of applications used in their environment that they approve or expect to see running. If an application is executed and does not appear on that list, it is considered an anomaly that will either be blocked or generate a notification to be sent to the security team. Application allowlisting gives a security team much tighter control over what is running in a given environment, and in addition to threat detection, it can be useful for managing network resources.

Application allowlisting is typically seen as an alternative to application blocklisting, which is a practice of creating a list of unwanted and/or blocked applications. From a security perspective, allowlisting is considered more effective, as by its nature it focuses on known applications that will be running in the environment. So, instead of trying to figure out all of the applications that need to be blocked (which really isn’t very feasible), the security team focuses on a known quantity of applications.

There is much more unknown when going the route of application blocklisting. Now, application allowlisting requires internal communication between the security team and the operational teams they support, but in the end the organization will have much better visibility (and tighter control) of the environment. And of course, application allowlisting will help prevent malware from entering and executing on network endpoints, and can also help in managing and reducing the use of network resources, improving overall network performance.

The challenges of application allowlisting

While allowlisting for better endpoint security sounds great in theory, it’s difficult to institute in practice. In order for allowlisting to be an effective option, the organization must have a strong and accurate asset inventory practice. The foundation of any security program is understanding what infrastructure, applications, and software are running in an environment, and which users have access to what systems. In order to begin allowlisting for security, you need to have a baseline of expected usage on the systems you want to protect. For instance, when planning to apply allowlisting to servers, you likely would monitor them for a period of time to understand typical usage patterns. Then you could analyze the results in order to create an appropriate allowlisting based off of that information.

This process may sound straightforward, but in most cases, it’s not. One reason this approach is so challenging is because the modern corporate environment changes so much with respect to technology, especially with the rise of cloud environments. With more employees having the ability to deploy infrastructure or software in the cloud without involving IT or corporate security, it becomes more challenging to track everything that’s being used or installed.

While it’s difficult for companies of any size, the problem becomes more pronounced the bigger and more complex the computing environment. When assets are spread out over different business units or groups—each of whom may or may not control their infrastructure or applications—getting the necessary visibility can become an overwhelming challenge.

Sound security fundamentals: The key to successful application allowlisting

If an enterprise has sound security fundamentals—such as an asset management program—in place, allowlisting and anomaly detection becomes easier to undertake. Before you implement a allowlist approach for security purposes, discuss these questions with your IT team:

• Can your IT team classify your servers and systems?
• Do they have a sense of what should be running on those hosts?
• What level of visibility do you have into your environment?

If you have a good sense of how your servers and environment should look, you’re in a good position to institute application allowlisting.

An anomaly detection example using Uptycs

In many organizations that run production applications, there are a set of servers that host the application(s), and the processes running on those endpoints are generally known. This is a perfect example of when baseline analysis and application allowlisting can be applied. In this scenario, osquery would be deployed to each of the application servers and those endpoints would be enrolled to the Uptycs platform.

From there the following steps would be performed:

Running process anomaly detection configuration

1. In Uptycs, tag the application servers in scope for this allowlisting effort.
1. In some organizations, all endpoints will be in scope. In others, there may be different baselines applied to different parts of the business.
2. If necessary, classify and tag the endpoints so the analysis of each set of systems can be performed appropriately. Auto tag rules can be used to tag assets based on the criteria that makes sense for the organization (i.e. host naming convention, IP address, OS, software installed, etc…)
2. After one week of data collection (which will include software installed and processes running), create a lookup table that will maintain the list of processes running on the set of tagged endpoints over that period of time. Don’t forget to review the lookup table to ensure the list of processes doesn’t include anything you deem unauthorized.
3. Customize an event/alert rule to use the newly created lookup table as the baseline to compare processes against for ongoing monitoring and detection.
4. Configure an alert notification for the appropriate team using the appropriate means of communication (e-mail, Slack, PagerDuty, HTTP webhook, etc…).

Keep learning with these detection-related resources

Allowlisting is just one approach to threat detection and endpoint security. If you’re interested in learning more about threat detection, we encourage you to take a look at the following on-demand webinars:

• On-Demand Webinar: Malware Detection with YARA and osquery
• On-Demand Webinar: Breach -> ATT&CK -> Osquery: Learning from Breach Reports to Improve Cross-platform Endpoint Monitoring