Log4j 2 CVE-2021-44228 | Software Architect’s Solution

Contents

What is Log4j2 CVE-2021-4428?
What is Vulnerable and Which Version Has The Fix?
What Does a Compromised Application Do?
What Inventory Should I Gather?
What Information Am I Gathering?
How Am I Fixing This?
What Do I Look For?
I Removed JndiLookup.class. How Do I Validate?
I Upgraded to a Fixed Version. How Do I Validate?
Build Process?
What Can My Other Teams Do?
What Else?

Log4j has lit the internet on fire. Understanding, scoping and assessing the situation due to log4j has been vital to remediation and finding a solution to this pervasive problem. In the last 2 weeks, I’ve had the good fortune to work with a talented team to address the problem for our own deployment and to help our customers. This posting is a generic summary of my notes, observations and actions used towards addressing the problem.

In a follow-up post, I’ll summarize how exactly we addressed the problem at Uptycs.

What is Log4j2 CVE-2021-44228?

One of the most severe security vulnerability in recent times
Only impacts Java application which are using a very popular logging library (many)
Application does not need to be Internet facing. If user provided input makes its way to the application and is processed by the logging library, it may be vulnerable
A well crafted string sent as a part of HTTP header(s) or the HTTP body, if it reaches the logging library can make the application vulnerable

What is Vulnerable and Which Version Has the Fix?

log4j (1.x) is a legacy library. Even though it has its own vulnerabilities, none as severe as this
log4j-core jar file from log4j2 library has the vulnerable class
log4j-core-2.0-beta9.jar or later versions are vulnerable
log4j-core-2.12.4.jar (for JDK7) and log4j-core-2.17.1.jar (for JDK8 and later) have the fix

What Does A Compromised Application Do?

Pretty much anything is possible
Exfiltrate secrets and other sensitive data
Move laterally to other parts of the infrastructure
Ransomware/spyware/... attacks
Exploit does not necessarily create additional processes
Initial exploit instructions may be downloaded from the Internet by the vulnerable Java application

What Inventory Should I Gather?

Dev, staging, test, production, ...
Cloud VMs, containers
Serverless
Artifactory and other internal caches
Container registries
Blob stores (S3, etc) with build artifacts
Employee machines

What Information Am I Gathering?

Vendor software with Java
Looking for vulnerable Java applications
Looking for all log4j version 1 and 2 artifacts

How Am I Fixing This?

Leave log4j 1.x alone? Upgrade? What about third-party software with no option to upgrade to the latest log4j2? (lower priority)
Upgrade to 2.12.4 (JDK 7) or 2.17.1 (JDK 8 and above)
Add Environment variable (LOG4J_FORMAT_MSGh_NO_LOOKUPS=true) or JVM property (-Dlog4j2.formatMsgNoLookups=true)? This doesn’t seem to fix the issue completely. Doesn't look like the issue is fully addressed:
Delete class file? Where is this done? Post build or pre deployment? Post build is ideal

What Do I Look For?

Java processes
Containers with Java processes
JDK/JRE version
jar/ear/war files. Unfortunately java -jar can run any arbitrary file with any extension. Lets hope you don't do that
Do I have uber jars, shaded jars, jmod, ...
Do I have compressed jars etc?
Are the jars obfuscated?

I Removed JndiLookup.class. How Do I Validate?

For each Java process, if jps is available:

for i in `jps | grep -v Jps | awk '{print $1}'`; do

jcmd $i GC.class_histogram | grep -q JndiLookup && {

echo "PID $i MIGHT be vulnerable";

};

done

Basically make sure the class is not loaded by your application

I Upgraded To A Fixed Version. How Do I Validate?

Get inventory of all jar/war/war/jmod files opened by the process
For each file opened by the process, check if it is or contains vulnerable log4j2-core jar
Validate on on-going basis to avoid deploying vulnerable applications due to auto scaling, old build artifacts etc.

Build Process?

Scan container images for vulnerable version
Scan Java artifacts for vulnerable version

What Can My Other Teams Do?

If you have a Web Application Firewall (WAF), use it. Make sure you apply the following rules:

${.*${ or ${jndi or ${ctx strings matching HTTP headers or body payload

If you have EDR/XDR product, for non-patched assets/Java processes:

Monitor processes spawned by Java processes
Monitor egress traffic to Internet (if the process is not expected to reach out to Internet)

Now is the time to look at egress firewall/security group rules. If your Java processes are in a subnet that the process is not expected to reach out to Internet, add egress blocking
Double check your data back-ups and disaster recovery steps

What Else?

Apply download blocking and remove vulnerable versions from artifactory etc.
Delete container images that are vulnerable to prevent accidental deployment
Delete build artifacts that are vulnerable
Make sure employees are not able to push to container registries and internal build artifact repository
Clean maven/gradle caches from employee machines. In case there are privileged users who can update container registries or build artifacts from their machines