Skip to content
Try it Free Request Your Demo
    December 13, 2021

    Log4j/Log4Shell Vulnerability Scanning and Exploit Detection in Uptycs osquery

    The Log4Shell vulnerability is already being compared to Heartbleed and Shellshock in terms of scope and severity, as it exposes nearly every Internet service and enterprise to ransomware and other attacks. Uptycs customers can take the steps below to protect themselves. Additionally, Uptycs XDR detects CVE-2021-44228 vulnerability post-exploit payloads and post-exploitation activities through our behavioral threat detection and YARA process memory and file scan components.

    Note: Many of the below queries work only on the Uptycs version of osquery. We will be publishing another blog with the queries that will work on open source osquery. The lookup across the fleet and historical lookback queries require the Uptycs Security Analytics Platform.

    image

    image from govcert

     

    Get an Inventory of all your log4j jars file, location and versions

    Query for log4j jar in non containerized (Host) environments

    Get inventory of all non uber jars 

    -- All platforms: Detect log4j2 jars on the host based on current running Java processes
    -- NOTE: This only identifies processes that have log4j2 in the Java process current working directory or sub-directories
    WITH

    envs AS (
      -- Find host Java processes that have LOG4J_FORMAT_MSG_NO_LOOKUPS environment variable set to 'true'
      SELECT e.pid,
             CASE
                WHEN e.key = 'LOG4J_FORMAT_MSG_NO_LOOKUPS' AND e.value = 'true' THEN true
                ELSE false
             END AS env_exists
      FROM process_envs e
      JOIN processes p
      USING (pid)
      WHERE e.key = 'LOG4J_FORMAT_MSG_NO_LOOKUPS'
            AND p.is_container_process = 0
            AND (p.name = 'java' OR p.name = 'java.exe')),

    procs AS (
      -- Find host Java processes and also identify if log4j2.formatMsgNoLookups system property is set to true
      SELECT p.pid, p.cwd, p.cmdline, COALESCE(e.env_exists, false) AS env_exists,
             CASE
               WHEN p.cmdline LIKE '%-Dlog4j2.formatMsgNoLookups=true%' THEN true
               ELSE false
             END AS flag_exists
      FROM processes p
      LEFT JOIN envs e
      USING (pid)
      WHERE p.is_container_process = 0
            AND (p.name = 'java' OR p.name = 'java.exe')
            AND cwd != '/')

    -- Find all the Java processes with log4j2 jars in the process current working directory
    SELECT f.filename, f.path, p.*
    FROM   procs p
    JOIN   file f
    WHERE  f.path LIKE p.cwd || '/%%'
           AND f.filename LIKE 'log4j%.jar'
           AND f.filename NOT LIKE 'log4j-1.%';


    Get inventory of all uber jars

    -- Linux: Detect log4j2 jars that may be embedded in Uber jar/war/ear files based on the Java processes running on the host
    -- NOTE: This only identifies processes that have log4j2 in the Java process current working directory, sub-directories or uber jar/war/ear
    WITH

    envs AS (
      -- Find host Java processes that have LOG4J_FORMAT_MSG_NO_LOOKUPS environment variable set to 'true'
      SELECT e.pid,
             CASE
                WHEN e.key = 'LOG4J_FORMAT_MSG_NO_LOOKUPS' AND e.value = 'true' THEN true
                ELSE false
             END AS env_exists
      FROM process_envs e
      JOIN processes p
      USING (pid)
      WHERE e.key = 'LOG4J_FORMAT_MSG_NO_LOOKUPS'
            AND p.is_container_process = 0
            AND p.name = 'java'),

    procs AS (
      -- Find host Java processes and also identify if log4j2.formatMsgNoLookups system property is set to true
      SELECT p.pid, p.cwd, p.cmdline, COALESCE(e.env_exists, false) AS env_exists,
             CASE
               WHEN p.cmdline LIKE '%-Dlog4j2.formatMsgNoLookups=true%' THEN true
               ELSE false
             END AS flag_exists
      FROM processes p
      LEFT JOIN envs e
      USING (pid)
      WHERE p.is_container_process = 0
            AND p.name = 'java'
            AND cwd != '/'),

    dirs AS (
      -- Find all distinct directories/sub-directories where jar/war/ear reside based on the Java process current working directory
      SELECT DISTINCT f.directory, p.*
      FROM   procs p
      JOIN   file f
      WHERE  f.path LIKE p.cwd || '/%%'
             AND (f.filename LIKE '%.jar' OR f.filename LIKE '%.ear' OR f.filename LIKE '%.war')),

    packages AS (
      -- Scan inside all jar/war/ear files found and look for log4j2 group ID ('org.apache.logging.log4j')
      SELECT *
      FROM java_packages j
      JOIN dirs
      USING (directory)
      WHERE group_id = 'org.apache.logging.log4j')

    -- Get all the host Java artifact details along with process details
    SELECT * FROM packages;


    Query for log4j jar in containerized environments

     

    Get inventory of all non uber jars

    -- Linux/macOS: Detect log4j2 jars in the containers based on current running Java containers
    -- NOTE: This only identifies processes that have log4j2 in the Java process current working directory or sub-directories
    WITH

    envs AS (
      -- Find container Java processes that have LOG4J_FORMAT_MSG_NO_LOOKUPS environment variable set to 'true'
      SELECT e.pid,
             CASE
                WHEN e.key = 'LOG4J_FORMAT_MSG_NO_LOOKUPS' AND e.value = 'true' THEN true
                ELSE false
             END AS env_exists
      FROM process_envs e
      JOIN processes p
      USING (pid)
      WHERE e.key = 'LOG4J_FORMAT_MSG_NO_LOOKUPS'
            AND p.is_container_process = 1
            AND (p.name = 'java' OR p.name = 'java.exe')),

    procs AS (
      -- Find container Java processes and also identify if log4j2.formatMsgNoLookups system property is set to true
      SELECT c.id as container_id, c.image, c.image_id, p.pid, p.cwd, p.cmdline, COALESCE(e.env_exists, false) AS env_exists,
             CASE
               WHEN p.cmdline LIKE '%-Dlog4j2.formatMsgNoLookups=true%' THEN true
               ELSE false
             END AS flag_exists
      FROM processes p, docker_containers c
      LEFT JOIN envs e
      USING (pid)
      WHERE p.is_container_process = 1
            AND (p.name = 'java' OR p.name = 'java.exe')
            AND p.pid_ns = c.pid_namespace)

    -- Find all the Java processes with log4j2 jars in the process current working directory
    SELECT f.filename, f.path, p.*
    FROM   procs p
    JOIN   file f
    WHERE  f.system_type = 'docker_container'
           AND f.system_id = p.container_id
           AND f.path LIKE p.cwd || '/%%'
           AND f.filename LIKE 'log4j%.jar'
           AND f.filename NOT LIKE 'log4j-1.%';


    Get inventory of all uber jars

    -- Linux: Detect log4j2 jars that may be embedded in Uber jar/war/ear files based on the Java processes running in containers
    -- NOTE: This only identifies processes that have log4j2 in the Java process current working directory, sub-directories or uber jar/war/ear

    WITH

    envs AS (
      -- Find container Java processes that have LOG4J_FORMAT_MSG_NO_LOOKUPS environment variable set to 'true'
      SELECT e.pid,
             CASE
                WHEN e.key = 'LOG4J_FORMAT_MSG_NO_LOOKUPS' AND e.value = 'true' THEN true
                ELSE false
             END AS env_exists
      FROM process_envs e
      JOIN processes p
      USING (pid)
      WHERE e.key = 'LOG4J_FORMAT_MSG_NO_LOOKUPS'
            AND p.is_container_process = 1
            AND (p.name = 'java' OR p.name = 'java.exe')),

    procs AS (
      -- Find container Java processes and also identify if log4j2.formatMsgNoLookups system property is set to true
      SELECT c.id as container_id, c.image, c.image_id, p.pid, p.cwd, p.cmdline, COALESCE(e.env_exists, false) AS env_exists,
             CASE
               WHEN p.cmdline LIKE '%-Dlog4j2.formatMsgNoLookups=true%' THEN true
               ELSE false
             END AS flag_exists
      FROM processes p, docker_containers c
      LEFT JOIN envs e
      USING (pid)
      WHERE p.is_container_process = 1
            AND (p.name = 'java' OR p.name = 'java.exe')
            AND p.pid_ns = c.pid_namespace),

    dirs AS (
      -- Find all distinct directories/sub-directories where jar/war/ear files reside based on the Java process current working directory
      SELECT DISTINCT f.directory, p.*
      FROM   procs p
      JOIN   file f
      WHERE  f.system_type = 'docker_container'
             AND f.system_id = p.container_id
             AND f.path LIKE p.cwd || '/%%'
             AND (f.filename LIKE '%.jar' OR f.filename LIKE '%.ear' OR f.filename LIKE '%.war')),

    packages AS (
      -- Scan inside all jar/war/ear files found and look for log4j2 group ID ('org.apache.logging.log4j')
      SELECT *
      FROM java_packages
      JOIN dirs d
      USING (directory)
      WHERE system_type = 'docker_container'
            AND system_id = d.container_id
            AND group_id = 'org.apache.logging.log4j')

    -- Get all the container Java artifact details along with container Java process details
    SELECT * FROM packages;


    Inventory of jar files based on CLASSPATH

    This query looks for the CLASSPATH variable on command line and searches that CLASSPATH for log4j jar’s. In some environments the jar’s may be loaded from a folder other than CWD. This will also report jar’s from CWD if the CWD path is specified in the CLASSPATH. Depending on the CLASSPATH specified there may be an overlap with the jar’s reported in the previous queries.

     

    Inventory in non-containerized (Host) environment

    -- All platforms: Detect log4j2 jars on the host based Java processes command line
    -- NOTE: This only identifies processes that have log4j (any version) in command line classpath
    WITH

    envs AS (
      -- Find host Java processes that have LOG4J_FORMAT_MSG_NO_LOOKUPS environment variable set to 'true'
      SELECT e.pid,
             CASE
                WHEN e.key = 'LOG4J_FORMAT_MSG_NO_LOOKUPS' AND e.value = 'true' THEN true
                ELSE false
             END AS env_exists
      FROM process_envs e
      JOIN processes p
      USING (pid)
      WHERE e.key = 'LOG4J_FORMAT_MSG_NO_LOOKUPS'
            AND p.is_container_process = 0
            AND (p.name = 'java' OR p.name = 'java.exe'))

    -- Find host Java processes and also identify if log4j2.formatMsgNoLookups system property is set to true
    SELECT p.pid, p.cwd, p.cmdline, COALESCE(e.env_exists, false) AS env_exists,
           CASE
             WHEN p.cmdline LIKE '%-Dlog4j2.formatMsgNoLookups=true%' THEN true
             ELSE false
           END AS flag_exists
    FROM processes p
    LEFT JOIN envs e
    USING (pid)
    WHERE p.is_container_process = 0
          AND (p.name = 'java' OR p.name = 'java.exe')
          AND p.cmdline LIKE '%log4j%';


    Inventory in a containerized environment

    -- Linux/macOS: Detect log4j2 jars in the containers based on current running Java containers command line
    -- NOTE: This only identifies processes that have log4j (any version) in command line classpath
    WITH

    envs AS (
      -- Find container Java processes that have LOG4J_FORMAT_MSG_NO_LOOKUPS environment variable set to 'true'
      SELECT e.pid,
             CASE
                WHEN e.key = 'LOG4J_FORMAT_MSG_NO_LOOKUPS' AND e.value = 'true' THEN true
                ELSE false
             END AS env_exists
      FROM process_envs e
      JOIN processes p
      USING (pid)
      WHERE e.key = 'LOG4J_FORMAT_MSG_NO_LOOKUPS'
            AND p.is_container_process = 1
            AND (p.name = 'java' OR p.name = 'java.exe'))

    -- Find container Java processes and also identify if log4j2.formatMsgNoLookups system property is set to true
    SELECT c.id as container_id, c.image, c.image_id, p.pid, p.cwd, p.cmdline, COALESCE(e.env_exists, false) AS env_exists,
           CASE
             WHEN p.cmdline LIKE '%-Dlog4j2.formatMsgNoLookups=true%' THEN true
             ELSE false
           END AS flag_exists
    FROM processes p, docker_containers c
    LEFT JOIN envs e
    USING (pid)
    WHERE p.is_container_process = 1
          AND (p.name = 'java' OR p.name = 'java.exe')
          AND p.pid_ns = c.pid_namespace
          AND p.cmdline LIKE '%log4j%';

     

    Query to find if a process has loaded a log4j jar

    The following real-time query will find all processes that have opened the Log4j jar file. In a large fleet we recommend that a query pack be created with this query and periodically check if a process has loaded Log4j jar.

    SELECT *
    FROM process_open_files
    WHERE path LIKE '%log4j%.jar'
          AND system_type IN ('host', 'docker_container');

     

    Detect public IP addresses with java programs on a host reached out to in the past few days

    This query is run on the Uptycs console as a global query. The Flight Recorder capability of Uptycs makes this query possible. The number of days you can look back at the raw telemetry depends upon the license purchased, one week being the minimum. 

    This query will provide a list of IP addresses which could then be reverse looked up for domain name (may not be 100% accurate but valuable for this investigation). Identify the IP addresses you do not recognize. Dig deeper by looking at the individual socket_events record, which binary path, what was the command line, what was the ancestry of the process that made the external connection. Add other exclusions based on your particular environment.

    SELECT DISTINCT remote_address
    FROM socket_events
    WHERE
      -- Since log4j vul was disclosed
      upt_day > 20211209
      -- Uncomment to limit to hosts with a specific pattern
      -- AND upt_hostname LIKE 'hostname%'
      -- Restrict to Java processes
      AND exe_name = 'java'
      -- loopback
      AND remote_address NOT LIKE '127.%'
      -- LAN: 10.x
      AND remote_address NOT LIKE '0000:0000:0000:0000:0000:ffff:0a%'
      -- AWS metadata: 169.254.169.254
      AND remote_address != '0000:0000:0000:0000:0000:ffff:a9fe:a9fe'

    Download the list of IP addresses, edit the downloaded CSV file to remove header line and the quotes and perform a reverse lookup using the following command

    for i in `cat /tmp/download.csv`; do dig +noall +answer -x $i; done

     

    Using Yara Rules

    Using Yara Rules to detect Log4j exploit attempts

    The set of YARA rules are constantly evolving as the attackers figure out more ways to obfuscate the JNDI request. As you run into new YARA rules feel free to use them in the query below to detect if your infrastructure has been exploited. If the YARA scan returns a match it means there was an exploitation attempt.

    This query does the following:

    1. Finds all the java processes running on a host
    2. Finds all of the log files opened by the each of the java processes
    3. Scans the log file with the yara rules provided
    4. Returns a count of how many matches were found and what string matched.

    You can find the full Yara rules on our Github page.

     

    Addendum for Uptycs Customers

    The real-time queries listed above provide data at a point in time on a single host. For example, the inventory queries return data for a single host when the query ran, but a day later your inventory could be different. For ongoing reporting, create a query pack using the real-time query and assign the query pack to a tag of hosts—assigning to “all” hosts covers your entire fleet. Schedule the querypack to run periodically. The data is collected in the Uptycs Flight Recorder. You can then write alerts to be informed immediately of a YARA rule match and you could write reports to display the hosts/containers and the paths to the Log4j jar files.

    Uptycs Customer Success has already pushed the required query packs and reports to all of our customers. The association of the tag to the querypack can be done for each customer after consulting with the customer. If you are an Uptycs customer, please reach out cs@uptycs.com. Detailed instructions have been provided on the Uptycs support portal.

     

    Uptycs also helps with Log4j remediation efforts, helping you quickly inventory vulnerable systems and prioritize patch efforts.

     

    To see how Uptycs helps with Log4J remediation, check out our interactive tour.

     

    Uptycs Team

    Other posts you might be interested in