The Power of Looking Back: Scanning Historical Data with the Latest Threat Intelligence

Posted by Amit Malik on 1/30/19 9:59 AM
Amit Malik
Find me on:

With polymorphic malware, quick-turn domains and turn-on-a-dime attack tactics, most security professionals are looking for real-time intelligence to enable protection that is as close to zero-day as possible. Finding a threat anywhere around the globe and then immediately blasting out a definition or identifying an artifact is high on the cybersecurity wish list.

However, chasing zero-days and the latest exploits, exciting and urgent as they might be, are not the only productive results of incident investigation efforts. Some of the best investigation results can be achieved by reviewing historical records in light of the the latest threat intelligence data.

Detecting threats is difficult work. False positives need to be minimized so that security systems and professionals can focus on real threats. The process of separating the real from the fake is never perfect. Some real threats will be missed, and some false positives will not be filtered out.

The Uptycs Security Analytics Platform uses osquery to collect system state data from connected endpoints, then streams that data over secure TLS protocol, storing it in an exclusive instance. This uniquely allows for both real-time and historical querying. The historical data helps to recreate the state of any particular system(s) at any point in time with past activities. This is a powerful feature for incident or breach investigation, but there is also significant advantage in being able to scan historical infrastructure wide system data for malicious activities which may have been previously missed, essentially combining the value of the latest threat intelligence with a historical view. In a way, this is an application of the old adage that hindsight is 20-20.

Threat Intelligence is a broad term and consists of multiple things, but in this case we are talking specifically about uncovering malicious domains and IP addresses. With Uptycs, the entire process of conducting regular historical data scans is vastly simplified. Before we dig into some specific examples of how this is done, let’s first discuss the threat intelligence provided by Uptycs.

Threat Intelligence Pruning Process

At Uptycs, we collect over 5 million pieces of threat intelligence indicators from more than 40 publicly available sources. Before releasing it to customers, we normalize, categorize and remove possible false positives. At present, we bucket threats into seven categories:

  1. Malware: This category contains all of the domains and IP addresses that are associated with malware activities, i.e. trojans, botnets, downloaders, droppers, exploits, etc.
  2. Attack: This category contains all of the IP addresses that are associated with attack activity, such as login bruteforce, DoS/DDoS attacks, etc.
  3. Anonymizer: This category contains all of the domains and IP addresses that are associated with anonymization networks like onion/tor networks.
  4. Coinminer: This category contains all of the domains and IP addresses that are associated with crypto currencies, including pool services and websites.
  5. Phishing: This category contains all of the domains and IP addresses that are associated with phishing activities.
  6. Dga: This category contains all of the domains that are generated using domain generation algorithm.
  7. Nrd: This category contains all of the newly registered domains.

For every alert triggered during real-time scanning, we provide the category, description and threat intelligence source name for additional context around the alert as shown in figure 1.

   Example of real time alerting from the Uptycs platform powered by osquery.Real time alert from Uptycs platform

 

Scanning Historical System Data with Fresh Threat Intel

Now, how can we use the same threat intelligence provided by Uptycs on historical data? The answer is simple. Our entire threat intelligence database is exposed in the SQL table upt_threat_indicators. We can run a simple join query on the global historical data connecting process_open_sockets, dns_lookup_events or socket_events tables with the upt_threat_indicators table. For example, the query provided in the example below will provide all of the detection results from the process_open_sockets table available from December 28th to current day. .

SELECT ut.indicator, ut.category, ut.description, p.pid, p.remote_address, p.upt_hostname from upt_threat_indicators ut inner join process_open_sockets p on p.remote_address = ut.indicator where p.upt_day >= 20181228 limit 1000;

In this example, I’ve set a limit on the number of days I’m looking back. It is advisable to start narrow (maybe a weeks time frame, first) and then expand in increments as you have a better handle on the amount of data your query will return.

 Once we launch the above query, we will get the following results as shown in figure 2:

Looking back on process_open_sockets using historical data scan. Historical data scan on process_open_sockets

We can further improve the query to get results filtered to a specific threat category. For example, if we want results for the malware category only, we can update the query to the following:

SELECT ut.indicator, ut.category, ut.description, p.pid, p.remote_address, p.upt_hostname from upt_threat_indicators ut inner join process_open_sockets p on p.remote_address = ut.indicator where p.upt_day >= 20181228 and ut.category = 'malware' limit 1000;

Similarly, the same query can be used on the dns_lookup_events table.

SELECT ut.indicator, ut.category, ut.description, p.pid, p.question, p.upt_hostname from upt_threat_indicators ut inner join dns_lookup_events p on p.question = ut.indicator where p.upt_day >= 20181228 limit 1000;

Once we launch the above query, we will get the following results as shown in figure 3.

Examinging dns_lookup_event table using historical data scan.Historical data scan on dns_lookup_event table

 

Excluding Existing Triggered Alerts

The above queries provide results that include alerts generated during the real-time scanning. But let’s say that in the historical scan we only want to report on the detections that are new or were otherwise not detected during the real-time scanning because at the time we had no intel about the domain or IP address. A slightly more complex query, such as the following, can zero in on results not yet detected in alerts: .

SELECT ut.INDICATOR,

ut.category,

ut.description,

p.pid,

p.remote_address,

p.upt_hostname

FROM upt_threat_indicators ut

INNER JOIN process_open_sockets p

ON p.remote_address = ut.INDICATOR

JOIN upt_alerts u ON (p.remote_address <> u.value AND p.upt_asset_id = u.upt_asset_id

AND p.upt_day = CAST(date_format(alert_time, '%Y%m%d') AS INTEGER))

WHERE p.upt_day >= 20181228 and ut.category = 'malware' limit 1000; 

Once we launch the above query we will get the following results as shown in figure 4.

Looking Historical scan on process_open_sockets excluding real-time alerts

Scanning historical data with the latest threat intelligence is a powerful approach to uncover infections that were not otherwise detected originally in a real-time scan. All of the queries can be scheduled to run automatically in the platform on a daily basis, or any other required interval, to generate reports for analysts to conduct further investigation.

Acknowledgement: Thanks to my colleague Vibhor Kumar in helping me with SQL queries.

Join our free, on-demand Intro to Osquery security training course.

Topics: incident investigation, cloud security, osquery, Insider, continuous monitoring, TLS

Uptycs Blog | Cloud Security Trends and Analysis

Welcome! The Uptycs blog is for security professionals and osquery enthusiasts interested in exploring new ideas in cloud security. We hope you'll enjoy our blog enough to subscribe, share and comment.

Find Uptycs Everywhere

Subscribe for New Posts

Recommended Reads