Most modern organizations understand that the earlier you integrate security into the development process, the more secure the applications will be in production. For containerized workloads, securing the container image throughout the application life cycle is a critical part of security, but many organizations don’t even follow basic best practices for ensuring secure container images.
Container images are applications’ unit of execution, the envelope that code is packaged in at all stages of the life cycle. This makes it easier for organizations to shift security left, because ensuring that the container image is secure early in the life cycle reduces the risk of security problems in production.
But this does not mean organizations can ignore security later on. Containers are opaque, making it difficult to see inside. Attacks may exploit vulnerabilities that are not yet identified, accessing your environment in a way you have not anticipated. Container security requires adaptable processes and continual vigilance.
Ensuring container security, in five steps
Here are five best practices focused specifically on ensuring that the container image is as secure as possible throughout the application’s life cycle. These are not the end-all of security for containerized workloads, but rather the foundation of a strong security posture. Without secure container images, however, it’s not possible to have a truly secure application.
Here are the best practices to improve your container images’ security posture.
Embed image scanning at every stage of the life cycle
Image scanning isn’t something that can be done once and checked off the security list. Rather, organizations should incorporate image scanning whenever possible in the application life cycle. This includes:
- When the image is being built, within the CI/CD pipeline.
- When the image is in your repo.
- Continuously, when the image is running.
New CVEs are continuously being discovered and published, and an image that was compliant yesterday might have a vulnerability today. Continuously scanning images in the registries and at runtime increases protection from this drift. You can also control image execution through policies that block images with a particular severity of CVE, depending on the sensitivity of the environment it will run in. In Kubernetes, this can be accomplished by leveraging capabilities such as the admission controller.
Inline scanning within your pipelines and registries can also provide better security and control. This consists of scanning locally in your environment without sharing the image contents or registry credentials externally.
Do not run images as root
This summer, Sysdig published the “2020 Container Security Snapshot,” and among the results, it found that 58% of images are running as root.
Running an image as root is sometimes the easiest way for developers to get an application to work, but it’s also risky, similar to locking your front door but leaving the window next to it wide open. If images are running as root, it also means execution of code will be root; attackers with root access can execute malicious processes inside the container. While kernel isolation prevents the attacker from using root privileges to access other parts of the infrastructure, it does not prevent an attacker from exploiting those services via the network (e.g., using Nmap to identify vulnerable services). An attacker with root access can also perform container breakouts by exploiting known kernel vulnerabilities such as runc.
By default, containers run with root permissions, so it is up to the organization to either change them manually or put automation systems in place to ensure containers don’t have root access.
Running as root is dangerous and should be avoided whenever possible, but it does happen. If risky configurations are detected at runtime, teams typically do not stop the containers because they are focused on deploying quickly. Instead, they run within a grace period and continuously monitor to ensure deviations from security policies are addressed.
Scan both OS and non-OS packages
A common misconception about container security is that only operating system packages need to be scanned. This is not true. Everything running in the container, whether it is open source or commercial, OS or non-OS, needs to be scanned, because a vulnerability could be found in both OS and non-OS packages and could be exploited by attackers.
Emphasizing the importance of also scanning non-OS packages, Sysdig found that 53% of non-OS packages have high or critical level severity vulnerabilities.
Be aware of provenance
Images that come from public repositories are going to be inherently less secure than images from private repos. For example, Docker Hub certifies less than 1% of its nearly 3 million hosted images.
Container images have many layers, and security problems can occur if any layer of the image contains vulnerabilities. When you take images from public repositories, you often know nothing about who created that image or where its components came from, so you have to be extra careful to ensure it doesn’t contain vulnerabilities.
According to the Sysdig survey, 40% of images are pulled from public sources.
On the other hand, a blanket ban on using public repositories is rarely a good idea. In fact, images from public repositories can be more secure because there are more eyeballs on them. The bottom line is that you should do a thorough scanning and analysis of all container images, especially those that come from public repositories. When pulling an image from a public repository, it’s also important to pay attention to how many people are using it. An image that’s been pulled three times is going to be much riskier than one that has been used 20 million times.
Regardless of where the image comes from, it’s important to be disciplined about tagging and versioning.
Keep images as small as possible
When it comes to security, the smaller the image, the better. There are fewer opportunities to exploit the image when the image surface layer is smaller. From a security standpoint, the ideal situation is images that run one single process. While that will rarely be the case, it should be the goal.
As an organization, work on stripping images of anything they don’t need. Make sure that, if possible, you use static linking and don’t have too many dynamic libraries or config files. The goal is to reduce the attack surface as much as possible without compromising the ability to get information about the container during an incident, or impeding the container from running successfully.
The ultimate goal: Secure container images
Secure container images are a necessary, but not sufficient, component of a strong security posture in containerized applications. If the container images aren’t as secure as possible, the other tools available to secure cloud-native applications, such as access controls and networking policies, won’t be enough to keep malicious actors from doing damage. Unfortunately, most organizations don’t pay enough attention to securing their container images. Just taking the simple steps outlined above will improve your organization’s security posture.