DIE is the notion that an immutable workload should not change during runtime; therefore, any observed change is potentially evident of malicious activity, also commonly referred to as Drift. Container Drift Detection provides an easy way to prevent attacks at runtime by simply following security best practices of immutability and ensuring containers aren’t modified after deployment in production.
Getting ahead of drift in container security
According to the Sysdig 2024 Cloud-Native Security & Usage Report, approximately 25% of Kubernetes users receive alerts on drift behavior. On the other hand, about 4% of teams are fully leveraging drift control policies by automatically blocking unexpected executions. In order to prevent drift, you need to be able to detect drift in real-time. And that’s where Falco’s rich system call collection and analysis is required. We will highlight how Falco rules can detect drift in real time, and provide some practical drift control advice.
Container drift detection when files are open and written
This Falco rule is rather rudimentary, but it still achieves its intended purpose. It looks for the following event types listed – open, openat, openat2, creat. This works, but it relies on fairly ambiguous kernel signals, and therefore only works with Falco Engine version 6 or higher. The rule is enabled by default in the stable rules feed of the Falco Rules Maturity Framework.
- rule: Container Drift Detected (open+create)
desc: Detects new executables created within a container as a result of open+create.
condition: >
evt.type in (open,openat,openat2,creat)
and evt.rawres>=0
and evt.is_open_exec=true
and container
and not runc_writing_exec_fifo
and not runc_writing_var_lib_docker
and not user_known_container_drift_activities
enabled: false
output: Drift detected (open+create), new executable created in a container (filename=%evt.arg.filename name=%evt.arg.name mode=%evt.arg.mode evt_type=%evt.type user=%user.name user_uid=%user.uid user_loginuid=%user.loginuid process=%proc.name proc_exepath=%proc.exepath parent=%proc.pname command=%proc.cmdline terminal=%proc.tty %container.info)
priority: ERROR
tags: [maturity_sandbox, container, process, filesystem, mitre_execution, T1059]
Code language: Perl (perl)To see which Falco rules are in what status of the Falco Maturity Framework, check out this link.maturity_stable indicates that the rule has undergone thorough evaluation by experts with hands-on production experience. These practitioners have determined that the rules embody best practices and exhibit optimal robustness, making it more difficult for attackers to bypass Falco detection.
Container Drift Detection through chmod
In Unix and similar operating systems, the chmod command and system call are utilized to modify the access rights and specific mode flags (such as setuid, setgid, and sticky flags) for file system entities, including both files and directories.
- rule: Container Drift Detected (chmod)
desc: Detects when new executables are created in a container as a result of chmod.
condition: >
chmod
and container
and evt.rawres>=0
and ((evt.arg.mode contains "S_IXUSR") or
(evt.arg.mode contains "S_IXGRP") or
(evt.arg.mode contains "S_IXOTH"))
and not runc_writing_exec_fifo
and not runc_writing_var_lib_docker
and not user_known_container_drift_activities
enabled: false
output: Drift detected (chmod), new executable created in a container (filename=%evt.arg.filename name=%evt.arg.name mode=%evt.arg.mode evt_type=%evt.type user=%user.name user_uid=%user.uid user_loginuid=%user.loginuid process=%proc.name proc_exepath=%proc.exepath parent=%proc.pname command=%proc.cmdline terminal=%proc.tty %container.info)
priority: ERROR
tags: [maturity_sandbox, container, process, filesystem, mitre_execution, T1059]
Code language: Perl (perl)While this Falco rule can generate significant noise, chmod usage is frequently linked to dropping and executing malicious implants. The rule is therefore disabled by default and placed within the “Sandbox” rules feed of the maturity matrix, however, it can be fine-tuned to better work for your environment.
The newer rule “Drop and execute new binary in container” provides more precise detection of this TTP using unambiguous kernel signals. It is recommended to use the new rule. However, this rule might be more relevant for auditing if applicable in your environment, such as when chmod is used on files within the /tmp folder.
Detect drift when a new binary is dropped and executed
It’s ideal to detect if an executable not belonging to the base image of a container is being executed. The drop and execute pattern can be observed very often after an attacker gained an initial foothold. The process is_exe_upper_layer filter field only applies for container runtimes that use overlayFS as a union mount filesystem.
- rule: Drop and execute new binary in container
desc: Detects if an executable not belonging to a container base image is executed.
condition: >
spawned_process
and container
and proc.is_exe_upper_layer=true
and not container.image.repository in (known_drop_and_execute_containers)
output: Executing binary not part of base image (proc_exe=%proc.exe proc_sname=%proc.sname gparent=%proc.aname[2] proc_exe_ino_ctime=%proc.exe_ino.ctime proc_exe_ino_mtime=%proc.exe_ino.mtime proc_exe_ino_ctime_duration_proc_start=%proc.exe_ino.ctime_duration_proc_start proc_cwd=%proc.cwd container_start_ts=%container.start_ts evt_type=%evt.type user=%user.name user_uid=%user.uid user_loginuid=%user.loginuid process=%proc.name proc_exepath=%proc.exepath parent=%proc.pname command=%proc.cmdline terminal=%proc.tty exe_flags=%evt.arg.flags %container.info)
priority: CRITICAL
tags: [maturity_stable, container, process, mitre_persistence, TA0003, PCI_DSS_11.5.1]
Code language: PHP (php)Adopters can utilize the provided template list known_drop_and_execute_containers containing allowed container images known to execute binaries not included in their base image. Alternatively, you could exclude non-production namespaces in Kubernetes settings by adjusting the rule further. This helps reduce noise by applying application and environment-specific knowledge to this rule. Common anti-patterns include administrators or SREs performing ad-hoc debugging.
Enforcing Container Drift Prevention at Runtime
Detecting container drift in real time is critical in reducing the risk of data theft or credential access in running workloads. Activating preventive drift control measures in production should reduce the amount of potentially malicious events requiring incident response intervention by approximately 9%, according to Sysdig’s report. That’s where Falco Talon comes to the rescue.
Falco Talon is a response engine for managing threats in your Kubernetes environment. It enhances the solutions proposed by the Falco community with a no-code, tailor-made solution for Falco rules. With easy to configure response actions, you can prevent the indicators of compromise in milliseconds.
- action: Terminate Pod
actionner: kubernetes:terminate
parameters:
ignoreDaemonsets: false
ignoreStatefulsets: true
- rule: Drift Prevention in a Kubernetes Pod
match:
rules:
- Drop and execute new binary in container
actions:
- action: Terminate Pod
parameters:
gracePeriods: 2
Code language: JavaScript (javascript)As you can see in the above Talon Rule Drift Prevention in a Kubernetes Pod, we have configured a response actionner for the Falco rule Drop and execute new binary in container. So, when a user attempts to alter a running container, we can instantly and gracefully terminate the pod, upholding the cloud-native principle of immutability. It’s crucial to remember the DIE concept here. Regular modifications during runtime, if not aligned with DIE, could lead to alert overload or significant system disruptions due to frequent workload interruptions when drift prevention is enabled.
If you do not intend to shutdown the workload in response to container drift detections, you could alternatively choose to run a shell script within the container to remove the recently dropped binary. Or, you could enforce a Kubernetes network policy to isolate the network requests from a suspected C2 server.
Conclusion
Drift control in running containers is not an optional feature, but rather a necessity when we talk about runtime security. When we look back at the DIE philosophy, we need a real-time approach, as seen in Falco, to protect immutable cloud-native workloads in Kubernetes. By leveraging Falco rules to monitor for unauthorized changes, such as file modifications or unexpected binary executions, organizations can detect and automatically mitigate potential security breaches through Falco Talon. This proactive approach to container security, emphasizing immutability and continuous surveillance, not only fortifies defenses against malicious activities but also aligns with best practices for maintaining the integrity and security of modern cloud-native applications.
Moreover, the adaptability of Falco’s rules to specific operational environments, through customization and the application of context-aware filters, enhances their effectiveness while minimizing false positives. This tailored approach ensures that security measures are both stringent and relevant, avoiding unnecessary alerts that could lead to alert fatigue among security teams. The journey towards a secure containerized environment is ongoing and requires vigilance, collaboration, and a commitment to security best practices.