Tales from the Kernel Parameter Side

Users live in the sunlit world of what they believe to be reality. But, there is, unseen by most, an underworld. A place that is just as real, but not as brightly lit. The Kernel Parameter side (apologies to George Romero).

Kernel parameters aren’t really that scary in actuality, but they can be a dark and cobweb-filled corner of the Linux world. Kernel parameters are the means by which we can pass parameters to the Linux (or Unix-like) kernel to control how that it behaves. By altering these parameters, we can control the behavior of quite a few different things, including memory, networking, and the filesystem, just to name a few.

There are a LOT of kernel parameters, somewhere in the vicinity of 2,000.

We may find differing parameters in the various flavors of Linux/Unix. Some of them are used or emphasized differently in specific environments, such as those we might find in the various cloud providers. In many cases, the sysctl utility, which we will discuss the use of shortly, is used to view or update these parameters. We may also find them in various parts of the filesystem, depending on the specific OS and distribution that we are working with.

Touring the kernel parameter landscape

We can find the kernel parameters stored in the filesystem as files under /proc/sys/. If we look at the directory structure here, we can see that it is extensive, relatively speaking.

Though the listing below is only two levels of directory structure deep, there are actually more directories under the ones we see here.

├── abi
├── debug
├── dev
│   ├── cdrom
│   ├── hpet
│   ├── mac_hid
│   ├── parport
│   ├── raid
│   ├── scsi
│   └── tty
├── fs
│   ├── binfmt_misc
│   ├── epoll
│   ├── fanotify
│   ├── inotify
│   ├── mqueue
│   ├── quota
│   └── verity
├── kernel
│   ├── firmware_config
│   ├── keys
│   ├── pty
│   ├── random
│   ├── seccomp
│   ├── usermodehelper
│   └── yama
├── net
│   ├── bridge
│   ├── core
│   ├── fan
│   ├── ipv4
│   ├── ipv6
│   ├── mptcp
│   ├── netfilter
│   └── unix
├── user
└── vmCode language: JavaScript (javascript)

If we look at the top-level structure under /proc/sys/, we can get at least a high level idea of what the different sets of parameters do:

• abi: parameters for execution domains and personalities
• debug: debug parameters
• dev: parameters for devices on the system
• fs: filesystem parameters
• kernel: parameters for operation of the kernel
• net: networking parameters
• user: user namespace parameters
• vm: memory management parameters

Each of the directories in this structure will have a set of files (or further directories) under it, one for each parameter, and each of them containing the value that the parameter in question holds. If we look at the structure under /proc/sys/dev/:

├── cdrom
│   ├── autoclose
│   ├── autoeject
│   ├── check_media
│   ├── debug
│   ├── info
│   └── lock
├── hpet
│   └── max-user-freq
├── mac_hid
│   ├── mouse_button2_keycode
│   ├── mouse_button3_keycode
│   └── mouse_button_emulation
├── parport
│   └── default
│       ├── spintime
│       └── timeslice
├── raid
│   ├── speed_limit_max
│   └── speed_limit_min
├── scsi
│   └── logging_level
└── tty
    └── ldisc_autoload
Code language: JavaScript (javascript)

We can see that we have a cdrom directory and, beneath it, a file called autoclose. This is the dev.cdrom.autoclose kernel parameter, which we will be working with shortly.

Some of these parameters are simple binary values, as is the case with dev.cdrom.autoclose. This parameter has a value of either 1 or 0. Some may have a wide range of numeric or even string values. Others, such as dev.cdrom.info, are odd.

CD-ROM information, Id: cdrom.c 3.20 2003/12/17
drive name:		sr1	sr0
drive speed:		1	1
drive # of slots:	1	1
Can close tray:		1	1
Can open tray:		1	1
Can lock tray:		1	1
Can change speed:	1	1
Can select disk:	0	0
Can read multisession:	1	1
Can read MCN:		1	1
Reports media changed:	1	1
Can play audio:		1	1
Can write CD-R:		1	1
Can write CD-RW:	1	1
Can read DVD:		1	1
Can write DVD-R:	1	1
Can write DVD-RAM:	1	1
Can read MRW:		0	0
Can write MRW:		0	0
Can write RAM:		0	0
Code language: JavaScript (javascript)

There is a great deal of information on the internet regarding the various kernel parameters and the impacts of changing them. One excellent resource for chasing down what a particular parameter does is Sysctl Explorer.

Working with kernel parameters

We need these tools with the following minimum versions to work through this demo:

• Ubuntu 22.04
• Falco 0.32.2 (install guide)

In these examples, we will be using Ubuntu 22.04 Jammy Jellyfish. Working with kernel parameters should generally be the same as the examples below on most distributions that are Debian-based, but may vary in strange and wonderful ways with others.

The easiest of these paths is to use the sysctl utility or add entries in /etc/sysctl.conf. Let’s try a few of these out. We’ll be interacting with the cdrom autoclose parameter as this is relatively innocuous and not even in use on many modern systems.

There are quite a few different methods that we can use to interact with kernel parameters on a live system. As with all things Linux, there is always a “one true” method for any given task. But here, we’ll be covering some of the more common means of doing so.

Viewing kernel parameters

To view a parameter and its value, we can use the sysctl utility to ask for it directly.

Sysctl is a tool specifically made to modify kernel parameters and is available on most flavors of Linux and Unix-like operating systems. In order to modify our parameter directly, we’ll need to know what the name of the parameter is in advance.

$ sudo sysctl dev.cdrom.autoclose
dev.cdrom.autoclose = 1Code language: JavaScript (javascript)

In the result returned for this, we can see that dev.cdrom.autoclose is set to 1, indicating that it is on and at its default value.

If we don’t immediately know the name of the parameter, we can get a list of all of them and their values with sysctl as well.

$ sudo sysctl -a
abi.vsyscall32 = 1
debug.exception-trace = 1
debug.kprobes-optimization = 1
dev.cdrom.autoclose = 1
<snip>Code language: JavaScript (javascript)

The parameter we’re working with is conveniently toward the beginning of the list, but there are a couple thousand or so parameters here. We can also, of course, just grep for what we’re looking for if we know roughly what we want.

$ sudo sysctl -a | grep cdrom
dev.cdrom.autoclose = 1
dev.cdrom.autoeject = 0
dev.cdrom.check_media = 0
dev.cdrom.debug = 0
<snip>Code language: JavaScript (javascript)

We can also look directly in the parameter file in the filesystem. In this case, in the file /proc/sys/dev/cdrom/autoclose.

$ cd /proc/sys/dev/cdrom
$ cat autoclose 
1
Code language: JavaScript (javascript)

Lastly, we can check in /etc/sysctl.conf and see if there is a parameter present here. There may be nothing in the file for our parameter at all, or it may be commented out. Most systems, by default, will not have many parameters in this file and they will all be commented out.

$ cat /etc/sysctl.conf 
#
# /etc/sysctl.conf - Configuration file for setting system variables
# See /etc/sysctl.d/ for additional system variables.
# See sysctl.conf (5) for information.
#
#kernel.domainname = example.com
# Uncomment the following to stop low-level messages on console
#kernel.printk = 3 4 1 3
<snip>Code language: JavaScript (javascript)

Again, we can simply grep in the file for the parameter that we are looking for to see if it is present at all.

cat /etc/sysctl.conf | grep cdromCode language: JavaScript (javascript)

We won’t see a return here in an unmodified system, as this parameter is not in /etc/sysctl.conf by default.

Updating kernel parameters using sysctl

We can easily edit the value of a parameter by using sysctl -w to write our desired value. This change will only persist until the next reboot of the system.

$ sudo sysctl -w dev.cdrom.autoclose=0
dev.cdrom.autoclose = 0Code language: JavaScript (javascript)

We can double check this change by querying the value again.

$ sudo sysctl dev.cdrom.autoclose
dev.cdrom.autoclose = 0Code language: JavaScript (javascript)

Updating kernel parameters by editing sysctl.conf

If we want to make a persistent change, we can add our parameter modification to /etc/sysctl.conf, like so.

$ sudo bash -c 'echo "dev.cdrom.autoclose = 1" >> /etc/sysctl.conf'Code language: JavaScript (javascript)

This is, however, not enough to accomplish the change entirely. If we check the state of the parameter now with sysctl, we will find that it has not changed.

$ sudo sysctl dev.cdrom.autoclose
dev.cdrom.autoclose = 0Code language: JavaScript (javascript)

Now, we need to load the changes from the files using sysctl -p. If we do not specify a file, sysctl will assume we want to load them from /etc/sysctl.conf, which is what we want. We can then check the parameter again and see the expected change.

$ sudo sysctl -p
$ sudo sysctl dev.cdrom.autoclose
dev.cdrom.autoclose = 1Code language: JavaScript (javascript)

Note:
There are a few other standard locations that sysctl will look for configuration files in:
/etc/sysctl.d/*.conf
/run/sysctl.d/*.conf
/usr/local/lib/sysctl.d/*.conf
/usr/lib/sysctl.d/*.conf
/lib/sysctl.d/*.conf
We won't get into these here, but the man page for sysctl.d does have extensive info on how these are used.Code language: JavaScript (javascript)

How to fail at directly editing the files under /proc/sys/

Even though the kernel parameter settings are stored in the individual files under /proc/sys/, we generally won’t be able to edit them directly. We can open them up and view the contents just fine, but if we make a change and try to write it to the file, even as root, we’ll get a variety of error messages, depending on the particular method that we try to take to do so:

• vi

  "autoclose" E667: Fsync failed

• nano

  [ Error writing autoclose: Invalid argument ]

• cat

  bash: autoclose: Permission denied

The items under /proc/ are a part of procfs, which is a virtual filesystem, so even if we were able to make changes here, they wouldn’t persist through a reboot. We can, however, watch these files for changes, which we’ll come back to shortly.

Which kernel parameters should we care about?

One of the harder issues when looking at kernel parameters is sorting out which of them we should care about. There are so many of them and such a wide range of settings that we can change.

How do we know which of these are important and which of them we should panic about when we see unexpected change?

This, as it turns out, is actually a really hard question to answer. To a certain extent, it depends on the specific OS, distribution, version, hardware, etc. that you are using. On the one hand, we might expect to rarely see any kernel parameter change in the environment, and it might be appropriate to ring the alarm bells any time we see one altered. On the other hand, we might have an environment where we see them change frequently. On some of the cloud platforms, kernel parameters are tweaked on instances for things like networking.

If we have our security measures turned up to super paranoid mode, we’re going to get a LOT of alerts – alerts which will soon be blissfully ignored because they are “always just noise,” which is clearly not desirable.

This being said, below is a non-exhaustive list of parameters that are generally considered “important.”

These should all be carefully tested before we randomly go rolling them out in a particular environment for monitoring or make any changes away from the defaults that they are set to. YMMV, IANAD (well..), etc…

dev.tty.ldisc_autoload	Disable loading line disciplines for unprivileged users
fs.protected_fifos fs.protected_hardlinks fs.protected_regular fs.protected_symlinks fs.suid_dumpable	Disable creating files in insecure areas Disable creating hardlinks for unprivileged users Disable creating files in insecure areas Disable creating symlinks for unprivileged users Disable core dumps for elevated processes
kernel.core_uses_pid kernel.ctrl-alt-del kernel.dmesg_restrict kernel.kptr_restrict kernel.modules_disabled kernel.perf_event_paranoid kernel.randomize_va_space kernel.sysrq kernel.unprivileged_bpf_disabled kernel.yama.ptrace_scope	Block USB devices Disable access to dmesg for unprivileged users Disable kexec to prevent kernel livepatching Restrict access to kernel logs Disable loading kernel modules Restrict use of performance events Address space randomization Harden debugging functionality No BPF for unprivileged users Limit the scope of ptrace
net.core.bpf_jit_harden	Harden the BPF JIT compiler
net.ipv4.conf.all.accept_redirects net.ipv4.conf.all.accept_source_route net.ipv4.conf.all.bootp_relay net.ipv4.conf.all.forwarding net.ipv4.conf.all.log_martians net.ipv4.conf.all.mc_forwarding net.ipv4.conf.all.proxy_arp net.ipv4.conf.all.rp_filter net.ipv4.conf.all.send_redirects net.ipv4.conf.default.accept_redirects net.ipv4.conf.default.accept_source_route net.ipv4.conf.default.log_martians net.ipv4.icmp_echo_ignore_broadcasts net.ipv4.icmp_ignore_bogus_error_responses net.ipv4.tcp_syncookies net.ipv4.tcp_timestamps	Disable ICMP redirect acceptance Disable source routing Disable BOOTP relay Disable IP forwarding Disable logging Martian packets Disable multicast routing Harden ARP Source route verification Disable ICMP redirect sending Disable ICMP redirect acceptance Disable source routing Disable logging Martian packets Ignore ICMP echo and timestamp requests from broadcasts Ignore bogus ICMP responses Enable syncookies Protect against time-wait assasination
net.ipv6.conf.all.accept_redirects net.ipv6.conf.all.accept_source_route net.ipv6.conf.default.accept_redirects net.ipv6.conf.default.accept_source_route	Protect against IP spoofing Protect against man-in-the-middle attacks Protect against IP spoofing Protect against man-in-the-middle attacks
vm.unprivileged_userfaultfd	Protect against use-after-free

There may also be other kernel parameters which, although not immediately sensitive, might be an indication of weirdness if we see them change on a system.

A good example of this is the vm.nr_hugepages parameter. This kernel parameter changes the memory chunk allocation so that memory lookups are faster, which can increase the efficiency of a cryptocurrency miner by 15-30%. Seeing this parameter unexpectedly change on a system may be a very good indicator that someone has gifted us with a miner.

What can the bad guys do with kernel parameters?

In short, quite a lot. Arguably, an attacker would need sufficient permissions on the system to alter the kernel parameters, or an attack that would allow them to subvert another privileged process to do so, but things can get quite a bit worse than this.

In the set of parameters we just discussed, even without digging too deeply, there are several that mitigate a variety of vulnerabilities related to network attacks. Disabling some or all of these intentionally could leave some large holes in the defenses of a system. These really aren’t the ugly parameters for an attacker to play with though.

To pull one out as an example, consider kernel.kexec_load_disabled. Unsetting this parameter can allow a replacement kernel to be loaded at runtime. Not just a replacement kernel, but any kernel, potentially even loading a completely unsigned one. Once we can modify the contents of the kernel at will, we are entirely beyond the reach of any protections that might be in place in the way of anti-malware, or really much of anything else.

Attacks against or using kexec have been a staple of breaking protections on mobile devices and game consoles for years. Pulling off an attack like this might be a bit more complicated than flipping a kernel parameter in the end, but this could give attackers a nice foothold for a starting point.

Tracking changes to kernel parameters

For purposes of changes being made at runtime, there are thankfully few places that we need to watch for changes being made to our systems.

If we want to watch for general changes being made to any parameter, we can monitor the use of the sysctl tool. The two primary uses of this tool that we would care about for this purpose would be the -w switch to write a parameter and the -p switch to load a file such as /etc/sysctl.conf.

Fortunately, whenever changes to kernel parameters are implemented, the individual file pertaining to the parameter being changed under /proc/sys/ is also altered. When we make a change to dev.cdrom.autoclose, the /proc/sys/dev/cdrom/autoclose file is written. This is something we can easily track also, and is a little more sure than tracking any particular mechanism for making changes. We can also be a bit more specific about which parameter changes we want to watch for when we are using this mechanism.

Let’s take a look at using Falco to do just this.

Building a Falco rule to track kernel parameter changes

After installing Falco, we’ll want to make a quick change to the Falco configuration file so we can easily watch the Falco logs. We need to edit /etc/falco/falco.yaml using sudo, and change the file output to true and the filename to /var/log/falco_events.log as shown below:

file_output:
  enabled: true
  keep_alive: false
  filename: /var/log/falco_events.logCode language: JavaScript (javascript)

Once this is done, we’ll open /etc/falco/falco_rules.local.yaml in an editor using sudo in order to start building our rule.

We will be building a modular rule in order to make it easier to update and modify in the future, and we will implement a list and a macro in addition to the rule. Check out the Falco documentation for more information on these concepts.

First, we will add a list called sensitive_kernel_parameter_files. This list will hold the specific filenames of the parameters we want to monitor. In this case, it’s /proc/sys/cdrom/autoclose.

- list: sensitive_kernel_parameter_files
   items: [/proc/sys/dev/cdrom/autoclose]Code language: JavaScript (javascript)

Next, we’ll add a macro called sensitive_kernel_parameters. In this, we will place a condition that looks for any file descriptor names that appear in the sensitive_kernel_parameter_names list we just created.

 - macro: sensitive_kernel_parameters
   condition: fd.name in (sensitive_kernel_parameter_files)Code language: JavaScript (javascript)

Lastly, we’ll create a rule with a condition that looks for modifications to kernel parameters called Kernel Parameter Modification. The fast and easy route for the condition in this rule might look something like the following:

condition: (open_write and (fd.name startswith "/proc/sys/" or fd.name startswith "/etc/sysctl"))Code language: JavaScript (javascript)

In this case we would watch for writes to any files under /proc/sys/ or /etc/sysctl. This would catch any changes that got made, right? Yes, it would.

Some cloud providers may change kernel parameters on instances in order to optimize them for their particular environment. It may seem like a good idea to set up a rule like “tell me every time any kernel parameter gets changed”, but we may potentially see a great deal of noise from doing so (that’s experience talking).

Instead, let’s make a condition that is a bit more specific. In this case, we’ll watch for files being written, which have a file descriptor name appearing in our sensitive_kernel_parameter_files list.

condition: (open_write and sensitive_kernel_parameters)Code language: JavaScript (javascript)

The final rule will look like so:

 - rule: Kernel Parameter Modification
   desc: Detect changes to sensitive kernel parameters. May be an indication of compromise.
   condition: (open_write and sensitive_kernel_parameters)
   output: >
     Sensitive kernel parameter was modified; possible unauthorized changes (user.name=%user.name user.loginuid=%user.loginuid proc.cmdline=%proc.cmdline parent=%proc.pname file=%fd.name container.id=%container.id container_name=%container.name evt.type=%evt.type evt.res=%evt.res proc.pid=%proc.pid proc.cwd=%proc.cwd proc.ppid=%proc.ppid proc.pcmdline=%proc.pcmdline proc.sid=%proc.sid proc.exepath=%proc.exepath user.uid=%user.uid user.loginname=%user.loginname group.gid=%group.gid group.name=%group.name image=%container.image.repository:%container.image.tag)
 priority: WARNINGCode language: JavaScript (javascript)

Finally, we need to restart the Falco service to put our updated rule into service.

sudo service falco restartCode language: JavaScript (javascript)

Testing our Falco rule

Let’s take everything for a test drive. We will want to use at least two terminals for this so we can easily keep track of everything that is going on.

First, we’ll set up a tail to follow anything that happens in the Falco log file. There will probably be some content here already from the Falco service ticking over, but we want to watch for new events being added.

In terminal 1:

tail -f /var/log/falco_events.logCode language: JavaScript (javascript)

Let’s first check the current state of the parameter. If we have followed all the steps above, dev.cdrom.autoclose is probably set to 1, but let’s take a peek anyway.

In terminal 2:

$ sudo sysctl dev.cdrom.autoclose
dev.cdrom.autoclose = 1Code language: JavaScript (javascript)

If it’s set to 0, that’s OK too. We can just reverse the sense of the command below if that’s the case. Let’s make the change:

$ sudo sysctl -w dev.cdrom.autoclose=0
dev.cdrom.autoclose = 0Code language: JavaScript (javascript)

In our tail of the Falco log in terminal 1, we should see the output of our kernel parameter change rule. Success!

11:19:22.673413934: Warning Kernel parameter was modified; possible unauthorized changes (user.name=root user.loginuid=1000 proc.cmdline=sysctl -w dev.cdrom.autoclose=0 parent=sudo file=/proc/sys/dev/cdrom/autoclose container.id=host container_name=host evt.type=openat evt.res=SUCCESS proc.pid=1447007 proc.cwd=/proc/sys/dev/cdrom/ proc.ppid=1447006 proc.pcmdline=sudo sysctl -w dev.cdrom.autoclose=0 proc.sid=5067 proc.exepath=/usr/sbin/sysctl user.uid=0 user.loginname=user group.gid=8390047166231478272 group.name=root image=<NA>:<NA>)Code language: JavaScript (javascript)

To expand what we have done here, we can go back and easily add whichever parameters we want to watch for modification to the sensitive_kernel_parameter_files list. We just need to know where in the filesystem the parameter lives in order to add it to the list.

Conclusion

Kernel parameters can be a bit scary at first glance, but are less intimidating than they appear to be once we get a bit better understanding of how they are structured and how they function. Once we have an idea of what the different sections of the directory structure under /proc/sys/ do, it becomes considerably easier to track down parameters and understand their purpose.

Armed with knowledge about what these parameters are and how we can interact with them, we can work on putting them to use in strengthening the security of our systems. As soon as we understand what the points are for monitoring kernel parameter changes, we can also help to protect ourselves from the impact of unexpected changes.

Falco could alert us about the modification of specific core parameters, avoiding the massive noise that this process causes.

---