What Is Detection Engineering? | Detection Engineering Explained

Detection engineering (DE) is a new approach to threat detection. More than just writing detection rules, DE is a process— applying systems thinking and engineering to more accurately detect threats. The goal is to create an automated system of threat detection which is customizable, flexible, repeatable, and produces high quality alerts for security teams to act upon.

DE is not yet a mature discipline with consistent methods and predictable results, but pioneers in the field are working towards this goal. There is not a lot written about the concept right now, no go-to standards or frameworks. It appears companies are developing their own approaches to incorporate the concept to improve their detection and response capabilities.

A concept called detection-as-code (DaC), first coined by Anton Chuvakin in 2020, is at the heart of DE, and points to the idea that detections should be treated as code. Essentially, it is about applying software engineering best practices to detections using modern agile CI/CD processes.

Benefits of Detection Engineering

Reduced mean-time-to-response through automation of detections.
Detections are tailored to the environment.
The process is structured and informs workflow.
Workflow is organized and teachable.
Code is version controlled, reused and modified as needed.
Peer review and testing are part of the process, to catch mistakes and gaps.
Coordination with external teams, like pentesting or threat hunting, is easier to manage.

Why Is Detection Engineering Important?

Security analysts are tasked with sifting through data and alerts using security tools like EDR, XDR, and SIEM to find malicious activity. False positives and false negatives are big problems for security teams. Too many false positives waste the time of the analysts, and false negatives let bad actors slip in undetected. The goal of any detection system is to reduce both and increase actionable insights for the security team to triage, remediate, and prevent in the future. Security analysts need to spend less time correlating signals, and more time fine-tuning their detections to gain high-fidelity alerts.

The point of detection engineering is to cover the edge cases that are not covered by your vendor’s built-in detection logic or in commercial and open-source threat intelligence feeds. Those rules and signatures are applicable to any environment. With DE, you want to find and fill in the detection gaps that are unique to your environment.

Detection Objective

Beginning with identifying threats relevant to an organization, threat modeling is the first step of Detection Engineering. The MITRE ATT&CK framework is a valuable tool that can direct an organization's detection objective by performing and providing a gap analysis, however the following questions remain key detection objective considerations:

What do I need to detect?
What threat actors, techniques, tools, etc., are relevant to us?
How can I demonstrate the relevance to the business?

Answering these questions can assist in building a formula for the threats an organization should be concerned about.

Detection Requirements

Identifying available log sources and determining what logs or data sources an organization is missing and necessary to detect the threats identified above is the next step, with identifying use cases, looking at vulnerability reports, and developing an understanding of the gaps in the organization’s security stance following.

This phase of determining the detection objective should answer the following questions:

How can I detect X?
What log or data sources do I need to do so?
What detection logic should I use?

Detection Implementation

As continuously tuning the specific pieces of content for detection that have been written for false positives and other nuances is crucial, using the answers to the questions above to consider a life cycle and how an organization will manage detection capabilities on an ongoing organization is the next step. Reviewing the detections that you’ve previously written while also tuning and deploying, below are the questions that should be asked during this process:

How can I automate detection?
Is this detection more suitable as a dashboard, saved search, report, or rule?

As detection engineering is cyclical in nature, having a supporting culture is a key element to ongoing success. Reduced mean time to detection and response of an incident is the return on investment of DE, with acute threat hunting safeguarding any surprise gaps not currently a part of the detection lifecycle.

Features of a Good Detection Engineering Process

Keep in mind that the DE process is more than detection-as-code. You will need both good tooling and rich detection content to work with. You will want to design a process which is repeatable and predictable. When designing your DE process you need to consider the before, during and after the writing of detection code.

A simple graphical view of the DE process from a SANS white paper is shown in Fig. 2. The process is cyclical, to be repeated continuously. Note that each stage is both dependent upon and contributing to the others. Each stage involves multiple tasks:

Detection Engineering Stage - This is where detections are written and a CI/CD process is followed for detections as code.
Detection Maintenance Stage - Test your detection system with pentesting, purple teaming and sandboxing. Discover new threat activity through threat hunting and setting up a honeypot to observe malicious behaviors. Document these findings and disseminate them to stakeholders. Create new or modify existing detection code based on these findings.
Threat Intelligence Stage - Both the beginning and the result of the DE process. Choose good threat intel resources and incorporate findings from internal tests and threat hunting.

The Role of Threat Intelligence in Detection Engineering

External threat intelligence correlated with internal endpoint/cloud/network visibility data are the source material detection engineers use to build good detection rules.

Threat Intelligence Sources

Threat intelligence platforms, feeds, and publishers are numerous. Some are branded, and some open-source. A security team will often subscribe to several of these resources, to stay abreast of new developments in the cyber threat landscape. Threat intelligence platforms (TIPs) offer threat data in machine-consumable form, so it can be integrated with existing security tools. Typically, these feeds provide static data points, like known bad IP addresses and malware hashes, so they alone will not provide enough context to detect behavioral patterns.

The other intelligence sources will inform your DE program, including information from industry information sharing and analysis centers (ISACs), government and law enforcement bulletins, and cybersecurity news feeds, especially if they contain more contextual data based on Tactics, Techniques, and Procedures (TTPs) described in the MITRE ATT&CK framework. Context about the attacker is invaluable, as the tools, tactics and techniques are the hardest for the attacker to change.

Indicators of Compromise

Before you start with DE, you need to know what indicators of compromise (IoCs) to look for. Threat intelligence sources are great for covering threats that are targeting any generic environment, but the point of DE is to fill in the gaps and build detections that cover your particular environment. So, how can you gather intelligence about threats that are unique to your situation? There are many IoCs, and not all are equal when it comes to detections.

According to a MITRE Technical Report: TTP-based Hunting, commonly used IoCs include static characteristics of malware like hashes, filenames, libraries, strings, or disk and memory forensics artifacts indicative of attack. Signature-based detection methods look for IoCs like these as indicators to trigger an alert. The problem with purely signature-based detection is that attackers have learned to morph these or change them frequently, and they do this with ease, evading detection. Also, signatures are based on known IoCs, which were previously detected, reported and disseminated, making them useless against novel threats.

Anomaly-based detection is another approach, which involves attempting to define normal activity on networks with large scale data collection and analysis, and then detecting abnormal behavior or anomalies in this data. Even with machine learning and large scale data analysis techniques, this method has been known to produce high false-positives and not enough context behind alerts.

The most effective indicators to use for detections are TTPs (tactics, techniques, and procedures) because these are the most difficult for an attacker to change. They would have to reinvent their methods, and they are restrained by the technologies they are targeting. The MITRE ATT&CK framework allows defenders to form hypotheses and hunt for novel threats based on adversary behavior, as well as use known TTPs to write detections. Importantly, TTPs may apply to both network and endpoint attack vectors. In Fig. 1 you can examine the famous “Pyramid of Pain” by David Bianco, often referenced when discussing the quality of different types of indicators for detections.

Generally, you want to aim for the top of the pyramid to detect behavior, because the higher the indicators are on the Pyramid of Pain (Fig 1), the more difficult it will be for the attacker to evade. Don’t discount the lower levels though, because the more information the better. Catching known hashes and bad IP addresses should be easy wins for your detection schema.

Proactive Threat Hunting

Threat hunting is a key part of threat intelligence. Threat hunters are specialized researchers who are often the source of new threat intelligence. They may use known TTPs and IoCs, along with their own expertise—like their knowledge of how hackers think, what they are after, etc.—to form a hypothesis as to what kind of attacks could happen in their systems.

They attempt to find evidence of nefarious activity which evades current threat detection and document their findings. That allows the security team to fine-tune detections to identify these techniques and procedures in the future. Upon documenting a new attack, threat hunters may report their findings to one or more threat intelligence publishers, benefitting the security community as a whole. Hence, the threat intelligence cycle renews continuously, as new discoveries are made and disseminated.

Pentesting & Honeypots

There are other proactive methods for seeking out new threat intel such as engaging in red teaming, purple teaming, pentesting, sandbox testing and using a honeypot. All of these help you better understand the threats targeting your systems and determine the efficacy of your detections. At least some of these exercises should be included in your DE process as a mechanism for gathering additional information about techniques and procedures used by attackers against your environment.

What Role Does an EDR/XDR Tool Play in DE?

In the SANS white paper: Detecting Malicious Activity in Large Enterprises, it is emphasized that absolute visibility across your environment (servers, containers, PCs, network, on-premises and in the cloud) is of utmost importance. You cannot secure what you cannot see. This is where a good EDR/XDR solution comes in.

An endpoint detection and response tool (EDR) or eXtended detection and response tool (XDR) serves as a security analytics engine. These can automatically make detections based on their programming and both external threat intelligence sources and internal knowledge of your environment. These detections may also be programmed to trigger an alert for the security team to act upon, or automatically respond to certain events like killing a process or shutting down a system.

However, the cyber threat landscape is always changing. While many detection rules may be included in your EDR/XDR platform, they are intended to serve as a strong base for most environments. Security teams cannot rely solely on the rules provided by others because there is no one-size-fits all rule set. Every environment is different and detections need to be tailored to the environment by the people living in it.

Ideally, you want your detection and response tool to be:

Compatible with on-premises, cloud, multi-cloud and hybrid environments
Flexible enough to adapt to an ever-changing threat landscape
Transparent in its function so that you can see how existing detections work and build off of that
Integrated with your existing security tools

Uptycs & DE

Uptycs security analytics platform offers multiple security solutions in one platform, including XDR capabilities, which can be used by security teams looking for an out-of-box solution that can be supplemented with customization on the fly. For example, see this case study where one large financial institution chose Uptycs precisely for the ability to customize their own rules and catch threats their current EDR was not catching. They needed a sophisticated and automated approach to YARA scanning at scale, and Uptycs provides a robust solution.

Plus, the Uptycs Threat Research Team is constantly on the hunt for new threats. They’ve found novel threats and contributed their findings to MITRE ATT&CK. In addition to seeking out novel threats, they remain informed by security news and threat feeds, and work to incorporate the latest threat intel into new rules for the Uptycs platform.

Importantly, Uptycs is an open platform. This means you can see how detections work, how they are scored and TTPs behind them. Use this information to both customize and create new detections for your organization.

Conclusion

If you have a threat research team who has the ability and the bandwidth to write all of your detections from scratch, more power to them. YARA and Sigma are the most popular languages used to write detection rules. YARA has become the industry-standard format for matching text and binary patterns in malware families.

An entire DE program can be designed in-house, and fully custom-built detections used with open-source security tooling. Obviously, this would be a time and resource intensive venture, but the results would be a detection system curated for your company’s environment and priorities.

Alternatively, you can use a vendor detection and response tool which is backed by a team of detection engineers, who continually update and improve the detections which are included in the platform. Then, supplement those built-in detections with your own customized rules, tailored to your environment.