How To Use Anomaly Detection for Application Whitelisting
The ability to identify anomalies within a computing environment is critical. Anomalies—events outside the norm—sometimes indicate security incidents, and usually prompt investigation. Whether they ultimately represent a security threat or not, anomalies are a leading indicator of something gone awry, which is why anomaly detection is a powerful tactic.
Many organizations are interested in applying the idea of anomaly detection as a key component of application whitelisting. But it’s difficult to do well, and its success depends in large part on the data that informs it. In this article we’ll explain the basics around application whitelisting, the challenges around implementing it, and show how security teams can use the Uptycs platform to successfully detect unwanted applications executing on endpoints.
What is Application Whitelisting?
The idea behind application whitelisting is simple: A security team creates a list of applications used in their environment that they approve or expect to see running. If an application is executed and does not appear on that list, it is considered an anomaly that will either be blocked or generate a notification to be sent to the security team. Application whitelisting gives a security team much tighter control over what is running in a given environment, and in addition to threat detection, it can be useful for managing network resources. (Tweet this!)
Application whitelisting is typically seen as an alternative to application blacklisting, which is a practice of creating a list of unwanted and/or blocked applications. From a security perspective, whitelisting is considered more effective, as by its nature it focuses on known applications that will be running in the environment. So, instead of trying to figure out all of the applications that need to be blocked (which really isn’t very feasible), the security team focuses on a known quantity of applications. There is much more unknown when going the route of application blacklisting. Now, application whitelisting requires internal communication between the security team and the operational teams they support, but in the end the organization will have much better visibility (and tighter control) of the environment. And of course, application whitelisting will help prevent malware from entering and executing on network endpoints, and can also help in managing and reducing the use of network resources, improving overall network performance.
The Challenges Of Application Whitelisting
While whitelisting for better endpoint security sounds great in theory, it’s difficult to institute in practice. In order for whitelisting to be an effective option, the organization must have a strong and accurate asset inventory practice. The foundation of any security program is understanding what infrastructure, applications, and software are running in an environment, and which users have access to what systems. In order to begin whitelisting for security, you need to have a baseline of expected usage on the systems you want to protect. For instance, when planning to apply whitelisting to servers, you likely would monitor them for a period of time to understand typical usage patterns. Then you could analyze the results in order to create an appropriate whitelist based off of that information.
This process may sound straightforward, but in most cases, it’s not. One reason this approach is so challenging is because the modern corporate environment changes so much with respect to technology, especially with the rise of cloud environments. With more employees having the ability to deploy infrastructure or software in the cloud without involving IT or corporate security, it becomes more challenging to track everything that’s being used or installed.
While it’s difficult for companies of any size, the problem becomes more pronounced the bigger and more complex the computing environment. When assets are spread out over different business units or groups—each of whom may or may not control their infrastructure or applications—getting the necessary visibility can become an overwhelming challenge.
Sound Security Fundamentals: The Key To Successful Application Whitelisting
If an enterprise has sound security fundamentals—such as an asset management program—in place, whitelisting and anomaly detection becomes easier to undertake. Before you implement a whitelist approach for security purposes, discuss these questions with your IT team:
- Can your IT team classify your servers and systems?
- Do they have a sense of what should be running on those hosts?
- What level of visibility do you have into your environment?
If you have a good sense of how your servers and environment should look, you’re in a good position to institute application whitelisting.
An Anomaly Detection Example Using Uptycs
In many organizations that run production applications, there are a set of servers that host the application(s), and the processes running on those endpoints are generally known. This is a perfect example of when baseline analysis and application whitelisting can be applied. In this scenario, osquery would be deployed to each of the application servers and those endpoints would be enrolled to the Uptycs platform. From there the following steps would be performed:
Running Process Anomaly Detection Configuration
- In Uptycs, tag the application servers in scope for this whitelisting effort.
- In some organizations, all endpoints will be in scope. In others, there may be different baselines applied to different parts of the business.
- If necessary, classify and tag the endpoints so the analysis of each set of systems can be performed appropriately. Auto tag rules can be used to tag assets based on the criteria that makes sense for the organization (i.e. host naming convention, IP address, OS, software installed, etc…)
- After one week of data collection (which will include software installed and processes running), create a lookup table that will maintain the list of processes running on the set of tagged endpoints over that period of time. Don’t forget to review the lookup table to ensure the list of processes doesn’t include anything you deem unauthorized.
- Customize an event/alert rule to use the newly created lookup table as the baseline to compare processes against for ongoing monitoring and detection.
- Configure an alert notification for the appropriate team using the appropriate means of communication (e-mail, Slack, PagerDuty, HTTP webhook, etc…).
Keep Learning With These Detection-Related Resources
Whitelisting is just one approach to threat detection and endpoint security. If you’re interested in learning more about threat detection, we encourage you to join our on-demand webinar: Network Threat Hunting with JA3 and osquery.
You can also learn more by viewing the following on-demand webinars:
- On-Demand Webinar: Malware Detection with YARA and osquery
- On-Demand Webinar: Breach -> ATT&CK -> Osquery: Learning from Breach Reports to Improve Cross-platform Endpoint Monitoring
And to see an example of the power of osquery for incident detection and investigation in action, watch a 15-minute demo of Incident Investigation with Uptycs.
Tagged as: whitelist security
There are no related posts
Subscribe for new posts
- Building Your Cyber Security Strategy: A Step-By-Step Guide
- Intro to Osquery: Frequently Asked Questions for Beginners
- Deploying Osquery at Scale: A Comprehensive List of Open Source Tools
- Osquery vs. OSSEC: Which Is Best for Linux Security in 2020?
- Windows Registry & Osquery: The Easy Way to Ensure Users are Secured