In the age of cloud computing, where more and more virtual hosts and servers are running some flavor of Linux distribution, attackers are continuously finding innovative ways to infiltrate cloud systems and exploit potential vulnerabilities. In fact, 91% of all malware infections were on Linux endpoints, according to a 2023 study by Elastic Security Labs. Another approach that poses a particularly grave danger to Linux VMs is the injection of a BPF (Berkeley Packet Filter) backdoor program into the host kernel.
This article explores why traditional Endpoint Detection and Response (EDR) solutions often struggle to detect and respond appropriately to kernel attacks, emphasizing the necessity of deep system call visibility to counter this alarming risk. We will conclude by emphasizing the need to secure every second in the cloud, by correlating host D&R activity like process behavior with cloud audit context for improved time to response in the cloud.
Understanding the Risk of BPF Backdoor Injection
The insertion of a BPF backdoor program into the operating system’s kernel is a menacing threat. The kernel, as the core of the operating system, is responsible for managing system resources and ensuring the integrity and security of the entire host/server. When an adversary successfully loads a BPF program into the kernel, they gain elevated privileges and the potential to manipulate host behavior without detection. This not only jeopardizes the confidentiality and integrity of sensitive data, but can also lead to unauthorized control of the host system.
Traditional EDR Shortcomings in Detecting Kernel Attacks
- Limited Visibility: Traditional EDR solutions rely on monitoring higher-level activities and processes, such as file and registry changes, network activity, user activity, and payload executions, often lacking deep visibility into kernel-level operations. This limitation makes them ill-equipped to detect or respond to threats that occur within the kernel.
- Ineffective Detection of Compiler Actions: While EDR solutions may claim to detect compiler actions, such as the compilation of malicious code, attackers can obfuscate these processes to evade detection. Techniques like Base64 encoding can be employed to camouflage compiler activities, rendering EDRs ineffective in identifying suspicious actions.
- Over Reliance on Policy Controls: Many EDR solutions primarily rely on protection policies that automatically enforce actions based on compiler activities. However, these policies can be disabled or misconfigured, leaving the system vulnerable to undetected threats. If an attacker successfully bypasses these controls, the EDR becomes powerless in preventing a breach.
The Need for Deep System Call Visibility
To effectively combat the threats posed by kernel attacks, it is imperative to adopt a deep, system call architectural view of the operating system. Sysdig, a leader in container security, has recognized the significance of this approach. By closely monitoring system calls at a granular level, it becomes possible to uncover even the most subtle deviations from normal behavior, allowing for real-time intrusion detection and response.
Sysdig’s Approach to Detecting BPF Backdoor Programs
Sysdig’s comprehensive threat detection capabilities are designed to combat kernel-level attacks. In the case of a BPF backdoor program injection, Sysdig can detect this activity by closely monitoring system calls, looking for irregularities and unauthorized modifications to the kernel. Even if an attacker employs obfuscation techniques, Sysdig’s deep system call visibility can unveil these deceptive actions.
Rule: eBPF Program Loaded into the Kernel
Description: This rule detects the loading of an eBPF program into the kernel. eBPF programs are extremely powerful, and as long as they conform to the constraints imposed by the eBPF verifier (they don’t cause a kernel panic), give near-arbitrary control over a target system.
As shown in the above screenshot, using the process tree, we can see that a parent powershell process (pwsh) is triggering a separate child process (bash) which is attempting to load the eBPF (extended Berkeley Packet Filter) program into the kernel. Thanks to this deep level of visibility, we aren’t solely notified of the suspicious behavior, but we can take remediation action quicker. We know where the bash commands are coming from, and can naturally take action on the parent program/script to remediate it from our system.
But what has this got to do with EDR detections?
The syscall argument provided in the rule output shows a base64-encoded shell command that decodes into a series of shell commands to load an eBPF program into the kernel and execute it. The process involves creating, compiling, and executing an eBPF program. Here is the full command listed under ‘Parent name and arguments’ of the Falco detection rule.
bash -c echo I2RlZmluZSBfR05VX1NPVVJDRQoKI2luY2x1ZGUgPHN0ZGlvLmg+CiNpbmNsdWRlIDxzdGRsaWIuaD4KI2luY2x1ZGUgPHVuaXN0ZC5oPgojaW5jbHVkZSA8c3lzL3N5c2NhbGwuaD4KI2luY2x1ZGUgPGZjbnRsLmg+CiNpbmNsdWRlIDxsaW51eC9icGYuaD4KCmludCBtYWluKGludCBhcmdjLCBjaGFyICoqYXJndikKewogICAgaW50IG47CiAgICBpbnQgYmZkLCBwZmQ7CiAgICBzdHJ1Y3QgYnBmX2luc24gKmluc247CiAgICB1bmlvbiBicGZfYXR0ciBhdHRyOwogICAgY2hhciBsb2dfYnVmWzQwOTZdOwogICAgY2hhciBidWZbXSA9ICJceDk1XHgwMFx4MDBceDAwXHgwMFx4MDBceDAwXHgwMCI7CgogICAgaW5zbiA9IChzdHJ1Y3QgYnBmX2luc24qKWJ1ZjsKICAgIGF0dHIucHJvZ190eXBlID0gQlBGX1BST0dfVFlQRV9LUFJPQkU7CiAgICBhdHRyLmluc25zID0gKHVuc2lnbmVkIGxvbmcpaW5zbjsKICAgIGF0dHIuaW5zbl9jbnQgPSBzaXplb2YoYnVmKSAvIHNpemVvZihzdHJ1Y3QgYnBmX2luc24pOwogICAgYXR0ci5saWNlbnNlID0gKHVuc2lnbmVkIGxvbmcpIkdQTCI7CiAgICBhdHRyLmxvZ19zaXplID0gc2l6ZW9mKGxvZ19idWYpOwogICAgYXR0ci5sb2dfYnVmID0gKHVuc2lnbmVkIGxvbmcpbG9nX2J1ZjsKICAgIGF0dHIubG9nX2xldmVsID0gMTsKICAgIGF0dHIua2Vybl92ZXJzaW9uID0gMjY0NjU2OwoKICAgIHBmZCA9IHN5c2NhbGwoU1lTX2JwZiwgQlBGX1BST0dfTE9BRCwgJmF0dHIsIHNpemVvZihhdHRyKSk7CiAgICBjbG9zZShwZmQpOwoKICAgIHJldHVybiAwOwp9Cg== | base64 -d > /tmp/prog.c; gcc /tmp/prog.c -o /tmp/prog && /tmp/progCode language: Perl (perl)Let’s break down the steps:
The bash -c ...: command starts a new Bash shell and executes the command that follows. This alone is not an issue for traditional security tools to detect.
This is where it gets difficult for an EDR. We see the echo … | base64 -d > /tmp/prog.c condition which decodes the base64-encoded string and saves it as a C source file named /tmp/prog.c. Once the C source code is encoded, an EDR will have a hard time detecting this behavior as suspicious since plenty of legitimate tokens are encoded in Base64 – it’s designed specifically for the purpose of making sensitive details hard to read. Of course, we can manually decode this ourselves with the below command:
<base64-encoded-value> | base64 -d
Once ended, the gcc /tmp/prog.c -o /tmp/prog command compiles the C source code (/tmp/prog.c) using the GCC compiler and creates an executable binary named /tmp/prog. The /tmp/prog program is finally able to execute the compiled binary, which contains the eBPF program. It’s possible that the EDR is able to quarantine the newly-compiled binary, but if it does, it is already lacking context into how the program got there in the first place. We would therefore have no visibility into how the malicious program was crafted on the host endpoint.
Assuming the EDR cannot detect the binary getting compiled due to incorrect configuration of the endpoint agent settings or false/negative detection from the detection engine, this sort of attack goes undetected potentially forever This provides easy access to network packets and the ability to take actions via programs written based on custom filters BEFORE they ever reach a (local) firewall.
Sysdig solves this problem of lost visibility through event enrichment across multiple data sources. Many EDRs simply do not ingest Kubernetes Audit logs, like Crowdstrike. These traditional tools are also unable to understand most real-time configuration changes via Cloudtrail audit logs alone. This fundamental lack of visibility in the cloud holds back security teams from responding quickly and with context.
Justifying the need for system call visibility
We have, of course, demonstrated the need for detecting when the actual BPF program is loaded into the kernel. But to improve overall cyberattack readiness, we need to detect all indicators of compromise associated with this host endpoint attack. We need to detect when potentially malicious scripts are encoded with Base64 before the adversary gets a chance to execute the program. True runtime security requires alerting on behavior that is undesirable in our environment, not just when a malicious payload is executed. By correlating context across process, container, host, Kubernetes, and Cloud, Sysdig improves time to response by telling you exactly where the encoded script was run from.
Regardless of how the script is encoded or its location within the system, Sysdig ensures consistent intrusion detection capabilities. Users have the flexibility to modify or craft their own detection rules to align with the specific behavior of their host system.
When there’s no legitimate rationale for encoding a shell script in your command line, you possess the means to take proactive measures well before potential adversaries or insider threats have an opportunity to inject a malicious program into the kernel, enabling them to maintain persistence and engage in further nefarious activities.
Conclusion
Attacks on the Host Kernel, such as the injection of a BPF backdoor program, pose a significant risk to the security machines running in the cloud. Traditional EDR solutions are ill-equipped to handle these threats due to their limited visibility and overreliance on policies. Sysdig’s deep system call architectural view for hosts and servers provides a crucial layer of defense, enabling the detection and response to even the most sophisticated kernel-level attacks, across Windows and Linux systems. In today’s constantly changing cloud-threat landscape, having this level of visibility is essential to prevent malicious intruders from accessing your sensitive data or abusing your cloud infrastructure for financial gain.