Tainted Kernels

Table of Contents

Introduction

A tainted kernel in Linux refers to a kernel that has been marked with one or more flags indicating that it is in a state that might affect its stability or functionality. Detecting tainted kernel module loads is crucial for ensuring system security and integrity, as malicious or unauthorized modules can compromise the kernel and lead to system vulnerabilities or unauthorized access. Source: elastic

Here are the main reasons a kernel might become tainted (list might not be exhaustive):

  • Proprietary Modules: When a proprietary (non-GPL) module is loaded into the kernel, the kernel is marked as tainted. This is because proprietary modules can affect the kernel in unpredictable ways, and developers cannot inspect the source code to diagnose issues.

  • Unsupported Modules: Loading modules that are not officially supported by the kernel’s maintainers.

  • Modules with Unsuitable Licensing: Loading modules with licenses that are considered incompatible with the kernel’s licensing can taint it.

  • Errors and Bugs: The kernel is marked as tainted if a serious error or bug occurs within the kernel, such as a kernel panic or an Oops (a non-fatal error).

kernel.org has an extended list of reasons a kernel becomes tainted.

Checking the state of the kernel

In the last blog post (Today I Learned - kernel.modules_disabled), we looked at how kernel modules are loaded and how to prevent them from being loaded by setting a kernel parameter. In our case, the kernel was marked as tainted because we loaded our custom DFIR kernel module. This information could be interesting from a defender’s standpoint, as a tainted kernel could indicate malicious activity.

We have several ways to determine whether the kernel is tainted. The first method is to search the kernel’s message buffer for the keyword “taint” - using the command dmesg to display the kernel messages.

dmesg

# dmesg  | grep taint
[  334.056326] dfir: loading out-of-tree module taints kernel.
[  334.056343] dfir: module verification failed: signature and/or required key missing - tainting kernel

We see the name of our kernel module (dfir) and the message that the loading of an unverified module tainted the kernel.

out-of-tree loaded modules

Look for unsigned and out-of-tree loaded modules. “out-of-tree” modules refer to kernel modules that are not included in the official Linux kernel source tree. These modules are developed, maintained, and distributed separately from the mainline kernel. Such out-of-tree modules are loaded with the command insmod, as we did with our own kernel module (see the latest blog post). Please note that specific device drivers for proprietary hardware can also taint a kernel, and that a tainted kernel must not always be a sign of an attacker or malicious activity.

# cat /proc/modules | grep OE
dfir 12288 0 - Live 0xffffffffc0c52000 (OE)

Again, our kernel module “dfir” is marked as out-of-tree.

Tainted flag(s)

The Linux kernel uses kernel taint flags to indicate that it has been modified or is running in a non-standard state. These flags help kernel developers and maintainers understand the conditions under which the kernel is running, which can be important for debugging and support purposes. Each flag represents a specific condition. A 0 in the file “/proc/sys/kernel/tainted” means the kernel is not tainted. However, in our case, the kernel is tainted:

# cat /proc/sys/kernel/tainted
12288

As this number alone is not really useful, let’s read what we can do instead on the official kernel.org documentation: The easiest way to decode that number is the script tools/debugging/kernel-chktaint, which your distribution might ship as part of a package called linux-tools or kernel-tools; if it doesn’t, you can download the script from git.kernel.org and execute it with sh kernel-chktaint [..]. Source: kernel.org

Following the steps from above:

# wget https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/tools/debugging/kernel-chktaint
# chmod +x kernel-chktaint 

# ./kernel-chktaint 
Kernel is "tainted" for the following reasons:
 * externally-built ('out-of-tree') module was loaded  (#12)
 * unsigned module was loaded (#13)
For a more detailed explanation of the various taint flags, see
 Documentation/admin-guide/tainted-kernels.rst in the Linux kernel sources
 or https://kernel.org/doc/html/latest/admin-guide/tainted-kernels.html
Raw taint value as int/string: 12288/'G           OE     '

And we found out that an out-of-tree and unsigned module was loaded. Unfortunately, this does not give us the module’s name, so we must revert to other methods outlined above or query the journal for taint messages.

Where’s Waldo?

So this is all fun and games until.. we want to find out where that module is located which tainted our kernel, or what the exact specifications of the module are, for an in-depth analysis. Let’s start with the basics:

We list all kernel modules that are currently loaded into the kernel by running the lsmod command. The first column is the name of the loaded module, the second the amount of memory the kernel module uses in kilobytes and the third the amount of dependencies to this module.

# lsmod | grep dfir
dfir 16384 0

This shows that the module is still loaded, but no further information is provided.. :( We can use the modinfo command to display detailed information about regular kernel modules:

# modinfo x_tables
filename:       /lib/modules/5.4.0-122-generic/kernel/net/netfilter/x_tables.ko
description:    {ip,ip6,arp,eb}_tables backend module
author:         Harald Welte <laforge@netfilter.org>
license:        GPL
srcversion:     0BAEF209167A3BD01B1ECFB
depends:        
retpoline:      Y
intree:         Y
name:           x_tables
vermagic:       5.4.0-122-generic SMP mod_unload modversions 
sig_id:         PKCS#7
signer:         Build time autogenerated kernel key
sig_key:        71:A4:C5:68:1B:4C:C6:E2:6D:11:37:59:EF:96:3F:B0:D9:4B:DD:F7
sig_hashalgo:   sha512
signature:      46:4B:33:2D:2F:4F:2F:F
[..]

But, modinfo can’t provide information about our module because modinfo only searches the standard system module directories (e.g., /lib/modules) for the kernel module file paths. (Thank you, Hal Pomeranz).

# modinfo dfir
modinfo: ERROR: Module dfir not found.

Pointing modinfo to our kernel module provides more information:

# modinfo dfir.ko
filename:       /root/dfir.ko
version:        0.1
description:    Simple Kernel Module for teaching Linux hardening techniques
author:         Stephan Berger
license:        GPL
srcversion:     C59846A6D64B564E0176416
depends:        
retpoline:      Y
name:           dfir
vermagic:       5.4.0-122-generic SMP mod_unload modversions

Continuing down the rabbit hole, we next check relevant entries in the journal.

# journalctl | grep dfir
Jul 11 09:44:28 ubuntu-s-1vcpu-512mb-10gb-fra1-01 kernel: dfir: loading out-of-tree module taints kernel.
Jul 11 09:44:28 ubuntu-s-1vcpu-512mb-10gb-fra1-01 kernel: dfir: module verification failed: signature and/or required key missing - tainting kernel

No news here. Last try: kallsyms. Kallsyms extracts all the non-stack symbols from a kernel and builds a data blob that can be linked into that kernel for use by debuggers. Source: kallsyms manpage

# cat /proc/kallsyms | grep -i dfir
ffffffffc0a7d024 r _note_6	[dfir]
ffffffffc0a7c000 t dfir_exit	[dfir]
ffffffffc0a7e000 d __this_module	[dfir]
ffffffffc0a7c000 t cleanup_module	[dfir]

This provides us with one of the function names (dfir_exit) but still does not provide us with more information about the module, such as the path from which it was loaded.

Conclusion

When a kernel is tainted, it is marked with one or more flags that indicate non-standard conditions or potential compromises. Understanding these flags is crucial for developers and system administrators in diagnosing and troubleshooting kernel issues, as well as identifying possible security breaches.

As we showed in the last section, tracing back from which location a kernel module was loaded is tricky (to impossible?). As security analyst, we must reside on different artefacts, like the .bash_history of the root user for executed commands (think of insmod), searching the file system for kernel files (with the ending .ko) or if it should be really your lucky day, auditing is enabled.