[s|l]trace - Linux Malware Analysis

Table of Contents


Craig Rowland, Founder and CEO of Sandfly Security, delivered a presentation titled Evasive Linux Malware at the Oslo Cold Incident Response Conference last year (Slides here, Presentation here), dissecting the notorious BPFDoor malware. In this post, we will analyze the BPFDoor backdoor only with the Linux utility strace, trying to get as much information as possible about the malware by tracing the executed syscalls from the binary. Swift assessments of malware samples like these can prove particularly beneficial for Incident Response teams in identifying Indicators of Compromise (IOC) for creating detection mechanisms or hunting purposes.


Figure 1: malpedia - elf.bpfdoor

I have assembled an archive containing the source code, the compiled binary, and the strace logs for your reference (password: infected). To begin, I will download the source code of BPFDoor from Pastebin, update the repositories, install gcc, compile the binary, and then set the executable flag on the compiled binary.

wget https://pastebin.com/raw/kmmJuuQP
apt update
apt install gcc
mv kmmJuuQP kmmJuuQP.c
gcc -o BPFDoor kmmJuuQP.c
chmod +x BPFDoor 

strace all the things

In the subsequent sections, we will utilize the output generated by strace. Following an excerpt from the strace man page:

In the simplest case strace runs the specified command until it exits. It intercepts and records the system calls which are called by a process and the signals which are received by a process. The name of each system call, its arguments and its return value are printed on standard error or to the file specified with the -o option.

We track the execution flow of BPFDoor with strace, using the -o option to record the output into distinct log files (a new log file will be created for each process, identifiable by the ProcessID or PID for short). The -ff option instructs strace to trace the child processes spawned by the binary, which will be useful in later stages, given that the binary undergoes multiple forks. The initial command for the analysis is as follows:

# strace -o log -ff ./BPFDoor

PID 6014 represents the initial process on my server (started with the command above). In the first line of the strace output, we observe the malware’s execution through the execve system call (the output is significantly condensed - please download the archive linked above to view the complete trace). Subsequently, the sample verifies the existence of the file /var/run/haldrund.pid. If the file exists, the sample terminates, indicating that the host is already infected, a fact corroborated by various other companies and researchers (see the link above to Malpedia, listing different articles about BPFDoor).

$ cat log.6014
execve("./BPFDoor", ["./BPFDoor"], 0x7ffde853bbb8 /* 12 vars */) = 0
access("/var/run/haldrund.pid", R_OK)   = -1 ENOENT (No such file or directory)
clone(child_stack=0x7f98927beff0, flags=CLONE_VM|CLONE_VFORK|SIGCHLD) = 6015
access("/var/run/haldrund.pid", R_OK)   = 0

In the listing above, we see the clone system call with a return value of 6015, representing the new process ID. Additionally, strace has generated a new log file for this specific process. In PID 6015, another execve syscall was invoked, passing the arguments /bin/rm -f /dev/shm/kdmtmpflush to /bin/sh.

$ cat log.6015
execve("/bin/sh", ["sh", "-c", "/bin/rm -f /dev/shm/kdmtmpflush;"...], 0x7ffc19e51608 /* 12 vars */) = 0

In PID 6016, the file /dev/shm/kdmtmpflush is deleted (see above).

$ cat log.6016
execve("/bin/rm", ["/bin/rm", "-f", "/dev/shm/kdmtmpflush"], 0x558e3e816888 /* 12 vars */) = 0

Following the cleanup, the sample copies itself to the location /dev/shm/kdmtmpflush, the same location previously cleared. We will proceed with the analysis outlined by Craig, documented here on the SandFly Security blog, aiming to correlate the evidence from the strace log(s) with the analysis results provided in the blog.

  1. Copy binary to the /dev/shm directory (Linux ramdisk).
  2. Rename and run itself as /dev/shm/kdmtmpflush.

Here is the relevant evidence from PID 6017 - Copy (Step 1) and Rename (Step 2):

$ cat log.6017
execve("/bin/cp", ["/bin/cp", "./BPFDoor", "/dev/shm/kdmtmpflush"], 0x558e3e816890 /* 12 vars */) = 0

Not explicitly highlighted by Craig, but permissions are adjusted to facilitate the execution of the newly copied binary (PID 6018).

$ cat log.6018
execve("/bin/chmod", ["/bin/chmod", "755", "/dev/shm/kdmtmpflush"], 0x558e3e816890 /* 12 vars */) = 0
fchmodat(AT_FDCWD, "/dev/shm/kdmtmpflush", 0755) = 0

The copying mechanism itself is quite interesting from a Linux internals perspective (PID 6017), demonstrating various system calls in action:

$ cat log.6017
stat("/dev/shm/kdmtmpflush", 0x7ffd7e118030) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "./BPFDoor", {st_mode=S_IFREG|0755, st_size=32608, ...}, 0) = 0
newfstatat(AT_FDCWD, "/dev/shm/kdmtmpflush", 0x7ffd7e117dc0, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "./BPFDoor", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0755, st_size=32608, ...}) = 0
openat(AT_FDCWD, "/dev/shm/kdmtmpflush", O_WRONLY|O_CREAT|O_EXCL, 0755) = 4
fstat(4, {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0
mmap(NULL, 139264, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8de68c0000
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220#\0\0\0\0\0\0"..., 131072) = 32608
write(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220#\0\0\0\0\0\0"..., 32608) = 32608
read(3, "", 131072)                     = 0
close(4)                                = 0
close(3)                                = 0

  1. Fork itself and run fork with “–init” flag which tells itself to execute secondary clean-up operations and go resident.

$ cat log.6019
execve("/dev/shm/kdmtmpflush", ["/dev/shm/kdmtmpflush", "--init"], 0x558e3e816868 /* 12 vars */) = 0

  1. The forked version timestomps the binary file /dev/shm/kdmtmpflush and initiates packet capture loop.

Timestomping is done with the utime() system call: The utime() system call changes the access and modification times of the inode specified by filename to the actime and modtime fields of times respectively. [1]

The function takes two parameters, const char *filename and const struct timeval times. The timeeval struct is defined as follows:

struct timeval {
    long tv_sec;        /* seconds */
    long tv_usec;       /* microseconds */

Examining our strace output, we observe the first parameter to utimes as the filename (/dev/shm/kdmtmpflush), and the second parameter as the timeeval struct containing seconds and microseconds. This action results in timestomping the binary to the value 2008-10-30 19:17:16.

$ cat log.6019
utimes("/dev/shm/kdmtmpflush", [{tv_sec=1225394236, tv_usec=0} /* 2008-10-30T19:17:16+0000 */, {tv_sec=1225394236, tv_usec=0} /* 2008-10-30T19:17:16+0000 */]) = 0

The activation of the filter occurs in a separate process (PID 6020).

On Linux, BPF is much simpler than on BSD. One does not have to worry about devices or anything like that. You simply create your filter code, send it to the kernel via the SO_ATTACH_FILTER option and if your filter code passes the kernel check on it, you then immediately begin filtering data on that socket. [2]

$ cat log.6020
socket(AF_PACKET, SOCK_RAW, htons(ETH_P_IP)) = 3
setsockopt(3, SOL_SOCKET, SO_ATTACH_FILTER, {len=30, filter=0x7fffa7c0bcb0}, 16) = 0

The system calls captured above resemble the code example from the kernel.org paper, as quoted above:

sock = socket(PF_PACKET, SOCK_RAW, htons(ETH_P_ALL));
if (sock < 0)
	/* ... bail out ... */

ret = setsockopt(sock, SOL_SOCKET, SO_ATTACH_FILTER, &bpf, sizeof(bpf));
if (ret < 0)
	/* ... bail out ... */

/* ... */

  1. The forked version creates a dropper at /var/run/haldrund.pid to mark it is resident and prevent multiple starts.

$ cat log.6020
openat(AT_FDCWD, "/var/run/haldrund.pid", O_WRONLY|O_CREAT, 0644) = 3

  1. The original execution process deletes the timestomped /dev/shm/kdmtmpflush and exits.

$ cat log.6021
execve("/bin/rm", ["/bin/rm", "-f", "/dev/shm/kdmtmpflush"], 0x558e3e816888 /* 12 vars */) = 0
unlinkat(AT_FDCWD, "/dev/shm/kdmtmpflush", 0) = -1 ENOENT (No such file or directory)
unlinkat(AT_FDCWD, "/dev/shm/kdmtmpflush", 0) = 0


Let’s consider a scenario where we lack prior knowledge about this specific malware sample. Tracing the execution would have revealed at least the following three key points:

  • /dev/shm/kdmtmpflush
  • /var/run/haldrund.pid
  • Creation of a socket

While the first two points could serve as Indicators of Compromise (IOCs) for searching other Linux servers for signs of a compromise, one aspect that needs to be addressed is under what name the sample would have been executed to conceal its identity. This information can be obtained using ltrace (as discussed in the next section).

ltrace FTW?

ltrace is a program that simply runs the specified command until it exits. It intercepts and records the dynamic library calls which are called by the executed process and the signals which are received by that process. [3]

ltrace will not work on latest versions of Ubuntu; read the details on the post Why “ltrace” does not work on new versions of Ubuntu? from Shlomi Boutnaru . We must compile our sample with different flags to get a traceable executable (see below).

# gcc kmmJuuQP.c -pg -ldl -no-pie

How can we determine which name the program selected from the array of possible names to camouflage itself? Let’s head to the ltrace log:

[pid 7162] srand(0x65afd911, 0, 0x7ffe6c1f9080, 0)                                                                                                 = 0
[pid 7162] rand(0x7f7535021240, 0x7f7535021740, 0x7f75350211c4, 0xffffffff)                                                                        = 0x37a64db0
[pid 7162] strcpy(0x408348, "dbus-daemon --system")                                                                                                = 0x408348

Which resembles the code here, taken from the source code from PasteBin:

strcpy(cfg.mask, self[rand()%10]);

self (the second parameter to strcpy, see above) is a pointer to the array of masquerading filenames and paths:

char *self[] = {
                "/sbin/udevd -d",
                "/sbin/mingetty /dev/tty7",
                "/usr/sbin/console-kit-daemon --no-daemon",
                "hald-addon-acpi: listening on acpi kernel interface /proc/acpi/event",
                "dbus-daemon --system",
                "pickup -l -t fifo -u",
                "avahi-daemon: chroot helper",
                "/sbin/auditd -n",

Double check: Searching for dbus-daemon –system (see the ltrace output above) on the list of running proceses on our infected server:

root@ltrace:~# ps aux | grep daemon
root        7163  0.0  0.0   2512   256 ?        Ss   15:19   0:00 dbus-daemon --system

Looks about right :)

root@ltrace:~# ls -l /proc/7163/exe 
lrwxrwxrwx 1 root root 0 Jan 23 15:44 /proc/7163/exe -> '/dev/shm/kdmtmpflush (deleted)'

Indeed, since we utilized two different tools to trace the malware’s execution, we may encounter two distinct names selected from the array of potential candidates. Nevertheless, this approach can still provide additional insights into the malware, revealing its attempt to camouflage itself.

Backdoor in Action

Sending back a ping

The following quotes are from the Elastic blog A peek behind the BPFDoor, demonstrating the capabilities of the malware:

  1. Observes incoming packets
  2. If a packet is observed that matches the BPF filters and contains the required data it is passed to the backdoor for processing

In our case, the required data to activate the backdoor is a password inside the payload sent to the infected host, hardcoded in the sample from Pastebin (justforfun or socket)

char hash[] = {0x6a, 0x75, 0x73, 0x74, 0x66, 0x6f, 0x72, 0x66, 0x75, 0x6e, 0x00}; // justforfun
char hash2[]= {0x73, 0x6f, 0x63, 0x6b, 0x65, 0x74, 0x00}; // socket

  1. It forks the current process again
  2. Changes the forked processes working directory to /
  3. Changes (spoofs) the name of the forked process to a hardcoded value
  4. Based on the password or existence of a password sent in the “magic packet” the backdoor provides a reverse shell, establishes a bind shell, or sends back a ping

The strace output below shows the ping in action; including step 13 (change the working directory to /) and step 14 (spoof the name of the process - /usr/libexec/postfix/master). The return value of the ping is “1”, as we can see in the source code, and also in the strace log:

$ cat log.11579 
chdir("/")                              = 0
setsid()                                = 11579
rt_sigaction(SIGHUP, {sa_handler=SIG_DFL, sa_mask=[HUP], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f9f8ef0fd60}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
prctl(PR_SET_NAME, "/usr/libexec/po"...) = 0
sendto(4, "1", 1, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("")}, 16) = -1 EINVAL (Invalid argument)
close(4)                                = 0
getpid()                                = 11579
exit_group(0)                           = ?
+++ exited with 0 +++

And the relevant code snippet from the source code, sending back the ping:

if ((s_len = sendto(sock, "1", 1, 0, (struct sockaddr *)&remote, sizeof(struct sockaddr))) < 0) {
    return -1;

The prct() system call conducts an operation on a process or thread. The argument PR_SET_NAME (first argument) set the name of the calling thread, using the value in the location pointed to by (char *) arg2. [4]