Log4j Vulnerability
Log4j vulnerability is all over the news in the IT world. Every team is scrambling to find out its impacts, verification process, and remedies. Log4j is one most popular logging frameworks in Java. Many Apache projects use this library for logging. Millions of devices (mostly the data center servers) are exposed to this exploit.
What is the issue:
Specific versions of log4j evaluate a part of the log message in the execution environment, allowing potentially vulnerable piece code to run remotely. Imagine that malicious users know specific messages (such as a contact us form) are logged using log4j. Now, a malicious user can send a piece code that could grant him high privilege access. For example, send a message to LDAP. Many articles have discussed the impact of this vulnerability on LDAP because it is a commonly used product for access control within IT infrastructure in large organizations. But, this vulnerability is undoubtedly not limited to LDAP.
Check out CVE details [https://nvd.nist.gov/vuln/detail/CVE-2021-44228] for impacted versions log4j.
Below is some code on how to reproduce the problem and resolve it. Also, how do you identify the spread of the scope within the data center and endpoints?
Demonstration of Vulnerability
Below the description of the environment in which we tested the vulnerability.
Operation System
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.3 LTS
Release: 20.04
Codename: focal
$ java -version
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (build 1.8.0_292-8u292-b10-0ubuntu1~20.04-b10)
OpenJDK 64-Bit Server VM (build 25.292-b10, mixed mode)
$ javac -version
javac 1.8.0_292
Download the dependent log4j jars which have the issue. Note the version of the log4j we are using 2.14.1.
wget https://repo1.maven.org/maven2/org/apache/logging/log4j/log4j-core/2.14.1/log4j-core-2.14.1.jar
wget https://repo1.maven.org/maven2/org/apache/logging/log4j/log4j-api/2.14.1/log4j-api-2.14.1.jar
Create a simple java file that will simply print the message in log. The message is passed by the user as command line argument.
TestApp.java
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.LogManager;
public class TestApp {
static Logger logger = LogManager.getLogger(TestApp.class);
public static void main(String... args) {
String message = args[0];
System.out.println("Argument: " + message);
logger.info(args[0]);
System.out.println("Main is exiting");
}
}
We need to log4j.xml (file name matters, it is case sensitive). This file controls logging behaviour of the log4j.
log4j2.xml
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO">
<Appenders>
<Console name="LogToConsole" target="SYSTEM_OUT">
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
</Console>
</Appenders>
<Loggers>
<Root level="info">
<AppenderRef ref="LogToConsole"/>
</Root>
</Loggers>
</Configuration>
Compile the TestApp.java
$ javac -cp log4j2.xml:log4j-api-2.14.1.jar:log4j-core-2.14.1.jar TestApp.java
Run the application. NOTE: put the message inside the single quotes. If you no argument you will get exception. Note that the second line (that contains INFO) Evaluates the input string and prints out the java version and the OS. This is root of the problem.
$ java -cp log4j2.xml:log4j-api-2.14.1.jar:log4j-core-2.14.1.jar:. TestApp '${java:version}/${java:os}'
Argument: ${java:version}/${java:os}
01:00:38.522 [main] INFO TestApp - Java version 1.8.0_292/Linux 5.4.0-81-generic unknown, architecture: amd64-64
Main is exiting
Resolution
There are a few know resolutions are there e.g. upgrade version of log4j, which is not always possible, because it maybe part of the database product, any version change need time consuming thorough regression testing and more importantly, it may require larger application change.
One short term quick solution is to set an environment variable.
Set the following environment variable to disable the feature (bug?)
export LOG4J_FORMAT_MSG_NO_LOOKUPS=true
Test the above execution once again. You should in the second line, input message is printed in the exact form.
$ java -cp log4j2.xml:log4j-api-2.14.1.jar:log4j-core-2.14.1.jar:. TestApp '${java:version}/${java:os}'
Argument: ${java:version}/${java:os}
01:00:59.029 [main] INFO TestApp - ${java:version}/${java:os}
Main is exiting
How do you find the which log4j.jar files are used by a java process. 3080037 is the PID in this example.
$ lsof -p 3080037 | grep log4j | grep jar
lsof: WARNING: can't stat() tracefs file system /sys/kernel/debug/tracing
Output information may be incomplete.
java 3080037 appuser mem REG 253,3 32522 32769279 /app/solr/server/lib/ext/log4j-web-2.11.2.jar
java 3080037 appuser mem REG 253,3 23239 32769278 /app/solr/server/lib/ext/log4j-slf4j-impl-2.11.2.jar
java 3080037 appuser mem REG 253,3 1629585 32769277 /app/solr/server/lib/ext/log4j-core-2.11.2.jar
java 3080037 appuser mem REG 253,3 266283 32769276 /app/solr/server/lib/ext/log4j-api-2.11.2.jar
java 3080037 appuser mem REG 253,3 64746 32769275 /app/solr/server/lib/ext/log4j-1.2-api-2.11.2.jar
java 3080037 appuser 54r REG 253,3 64746 32769275 /app/solr/server/lib/ext/log4j-1.2-api-2.11.2.jar
java 3080037 appuser 55r REG 253,3 266283 32769276 /app/solr/server/lib/ext/log4j-api-2.11.2.jar
java 3080037 appuser 56r REG 253,3 1629585 32769277 /app/solr/server/lib/ext/log4j-core-2.11.2.jar
java 3080037 appuser 57r REG 253,3 23239 32769278 /app/solr/server/lib/ext/log4j-slf4j-impl-2.11.2.jar
java 3080037 appuser 58r REG 253,3 32522 32769279 /app/solr/server/lib/ext/log4j-web-2.11.2.jar
Issues with the above command is if the log4j is embedded inside another jar file, for example SpringBoot, or Spark FAT jar, the above command cannot find jar file. You can unzip such FAT jar and inspect the version of the jar file.
Search for log4j files on the system
$ sudo find / -name "*log4j*.jar" | grep jar
/app/cassandra/lib/log4j-over-slf4j-1.7.25.jar
/app/zookeeper/lib/log4j-1.2.17.jar
/app/zookeeper/lib/slf4j-log4j12-1.7.25.jar
/app/kafka/libs/log4j-1.2.17.jar
/app/kafka/libs/slf4j-log4j12-1.7.28.jar
/app/kafka/libs/kafka-log4j-appender-2.4.1.jar
/app/spark-3.0.1-bin-hadoop3.2/jars/slf4j-log4j12-1.7.30.jar
/app/spark-3.0.1-bin-hadoop3.2/jars/log4j-1.2.17.jar
/app/solr/contrib/prometheus-exporter/lib/log4j-api-2.11.2.jar
/app/solr/contrib/prometheus-exporter/lib/log4j-slf4j-impl-2.11.2.jar
/app/solr/contrib/prometheus-exporter/lib/log4j-core-2.11.2.jar
/app/solr/server/lib/ext/log4j-1.2-api-2.11.2.jar
/app/solr/server/lib/ext/log4j-web-2.11.2.jar
/app/solr/server/lib/ext/log4j-api-2.11.2.jar
/app/solr/server/lib/ext/log4j-slf4j-impl-2.11.2.jar
/app/solr/server/lib/ext/log4j-core-2.11.2.jar
/app/solr/test/log4j-api-2.14.1.jar
/app/solr/test/log4j-core-2.14.1.jar
/app/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar
/app/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar
/app/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar
...
References:
https://fossa.com/blog/log4j-log4shell-zero-day-vulnerability-impact-fixes/
https://www.sophos.com/en-us/security-advisories/sophos-sa-20211210-log4j-rce
https://edition.cnn.com/2021/12/11/politics/dhs-log4j-software-flaw-warning/index.html