Image scanning for GitLab CI/CD

By Eduardo Mínguez - OCTOBER 12, 2022
Topics: Kubernetes & Container Security

SHARE:

content:

Image vulnerability scanning with Sysdig Secure Vulnerability scanning with GitLab CI/CD Creating a GitLab CI/CD pipeline Conclusions

Scanning a container image for vulnerabilities or misconfigurations on your GitLab CI/CD using Sysdig Secure is a straightforward process. This article demonstrates a step-by-step example of how to do it.


The following proof of content showcased how to leverage the sysdig-cli-scanner with GitLab CI/CD. Although possible, this procedure is not officially supported by Sysdig, so we recommend checking the documentation to adapt these steps to your environment.
This blog post is focused on the vulnerability scanner available since April 2022. If you are using the legacy scanner, see the official documentation for more information about it.


You can go straight to the pipeline definition here.

Image vulnerability scanning with Sysdig Secure

Image scanning allows DevOps teams to shift security left by detecting known vulnerabilities and validating container build configuration early in their pipelines before the containers are deployed in production, or images are pushed into any container registry. This allows detecting and fixing issues faster, avoids vulnerabilities in production or credential leaks, and improves the delivery to production time, all in a much more secure way.

The Sysdig Image Scanning process is based on policies that can be customized to include different rules, including ImageConfig checks (e.g., leakage of sensitive information) and checks for not just OS packages (rpm, deb, etc.), but also language specific packages and libraries (java, python, etc.).

Sysdig vulnerability scanning classifies images differently depending on where the scanning procedure is performed:

  • Pipeline: Before the runtime phase (in the developer workstation, in a CI/CD pipeline, etc.) performed by the sysdig-cli-scanner tool.
  • Runtime: When the image is running in the execution node and the scanning is performed by a Sysdig agent.

In this article, we will cover how to perform scanning on the pipeline step using GitLab CI/CD, as it is a best practice to adopt.

Running the scanner against a container image is as simple as running the sysdig-cli-scanner tool with a few flags (see the official documentation for more information), such as:

SECURE_API_TOKEN=<your-api-token> ./sysdig-cli-scanner --apiurl <sysdig-api-url> <image-name> --policy <my-policy>
Code language: JavaScript (javascript)

The image is scanned locally on the host where the tool is executed, on your laptop or a container running the pipeline, and only the scanning results are sent to the Sysdig Secure backend.

Vulnerability scanning with GitLab CI/CD

GitLab CI/CD is an open source continuous integration and delivery server integrated with the GitLab software development and collaboration platform.

Once you have configured GitLab CI/CD for your repo, every time a developer pushes a commit to the tracked repository branches, the pipeline scripts will be automatically triggered.

You can use these pipelines to automate many processes. Common tasks include QA testing, building software distribution artifacts (like container images or Linux packages), validating configuration, vulnerabilities, and compliance.

Image scanning has become a critical step in CI/CD workflows by introducing security earlier in the development process (security shift-left). Our workflow will build a container image, then it will locally scan the image using the sysdig-cli-scanner tool. The scan results will then be sent to Sysdig. If the scan evaluation fails, the workflow breaks, preventing the image from being uploaded into a registry.. Otherwise, the container image is pushed to the GitLab container registry.

Creating a GitLab CI/CD pipeline

The versions used in this example are:

  • Runner: 15.4.0~beta.5.gdefc7017 using Docker executor with image ruby:2.5.
  • Sysdig-cli-scanner version 1.2.9-rc, commit: e716ba6.

If you’re using the legacy scanner, the pipeline definition is different. See an example provided in the sysdiglabs/secure-inline-scan-example repository.

Prerequisites

The requirements for getting Sysdig Image Scanning up and running are straightforward:

  • A GitLab repository with administrative privileges.
  • A Sysdig Secure account where the results of the scans will be collected. You can request a free trial if you don’t have one.
  • A container Dockerfile ready to be built. You can fork and use our example, but using your own container is more fun!

Once you are ready, let’s move on!

Configure GitLab CI/CD in your repository

GitLab CI/CD is enabled by default on all new projects but if you did not activate GitLab CI/CD for your repository, the first step is making sure it is enabled. Navigate to your repository Settings, CI/CD section (https://gitlab.com/<user>/<repository>/edit), expand the “Visibility, project features, permissions” section, and make sure the “CI/CD” toggle is active:


Make sure the “Container registry” toggle is enabled as we will use it to store the container image as well.

Masked variables

For sensitive data such as the registry password or the API token, it is recommended to create masked variables instead in the repository Settings -> CI/CD -> Variables -> “Add variable.”

In this example, we will add the SYSDIG_SECURE_TOKEN to store the Sysdig API token required to query the Sysdig API and send the scan results. See the Sysdig official documentation for more information on how to get it.


It is worth mentioning that the variable should be masked to avoid potential security issues or leaks. See the official documentation for more information.

Container registry

We will leverage GitLab’s container registry to store the container image once the scan has been successfully completed.

There are a few special CI/CD variables to use the Container registry (CI_REGISTRY*) that are populated automatically by GitLab, so there is no need to specify them in our pipeline if we want to use it. Cool!

GitLab’s official documentation explains this in more detail, but the following is an example of the variables’ content once they are automatically populated:

  • CI_REGISTRY="registry.example.com"
  • CI_REGISTRY_IMAGE="registry.example.com/gitlab-org/gitlab-foss"
  • CI_REGISTRY_USER="gitlab-ci-token"
  • CI_REGISTRY_PASSWORD="[masked]"

Set up an image scanning GitLab CI/CD pipeline

GitLab CI/CD pipelines are defined by YAML files in the .gitlab-ci.yml file inside your repository by default. Pipelines can have variables and be broken into different steps, where every one can have different properties or run multiple commands/scripts.

Pipelines can be edited by using the Pipeline editor in the GitLab UI or just committing changes to the .gitlab-ci.yml file inside your repository.

Let’s see the pipeline definition in detail:

Variables and stages definition

variables:
  SYSDIG_SECURE_ENDPOINT: "https://eu1.app.sysdig.com"
  CI_IMAGE_TAG: "my-tag"
stages:
  - build
  - scan
  - push
Code language: Perl (perl)

We use a couple of variables to store the Sysdig API endpoint as well as the container image tag we want to use for the container image we are building. Also, we define the stages we are going to use to build, scan, and push the container image.

Build stage

image:build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - /kaniko/executor --dockerfile Dockerfile --destination $CI_REGISTRY_IMAGE:$CI_IMAGE_TAG --no-push --oci-layout-path $(pwd)/build/ --tarPath $(pwd)/build/$CI_IMAGE_TAG.tar
  artifacts:
    paths:
      - build/
    expire_in: 1 days
Code language: Perl (perl)

The image building step leverages the Kaniko project to build the container image using the instructions from your Dockerfile, and generates a new local image in the $(pwd)/build/$CI_IMAGE_TAG.tar file that will be scanned in the next step.

Scan stage

image:scan:
  stage: scan
  before_script:
    - export SECURE_API_TOKEN=$SYSDIG_SECURE_TOKEN
  script:
    - curl -LO https://download.sysdig.com/scanning/bin/sysdig-cli-scanner/$(curl -L -s https://download.sysdig.com/scanning/sysdig-cli-scanner/latest_version.txt)/linux/amd64/sysdig-cli-scanner
    - chmod +x ./sysdig-cli-scanner
    - ./sysdig-cli-scanner --console-log  --apiurl $SYSDIG_SECURE_ENDPOINT file://$(pwd)/build/$CI_IMAGE_TAG.tar
  artifacts:
    paths:
      - build/
    expire_in: 1 days
    when: always
  needs:
    - image:build
Code language: Perl (perl)

At this stage, we will be scanning the image for vulnerabilities and validating configuration, storing the results on the Sysdig backend. One of the benefits of Sysdig’s local scanning approach is that you don’t lose control over your images, as the image doesn’t need to be pushed to any registry or exposed externally in order to scan it. Instead, the scanning happens inside the runner and only the results are sent to Sysdig Secure.

The scanning process is as simple as downloading a binary file and executing it with a few parameters (including the SECURE_API_TOKEN environment variable from the SYSDIG_SECURE_TOKEN variable created before) against the container image built before.

Sysdig Secure will return an error code for this stage if the image contains any of the stop conditions configured in your policy (e.g., a critical vulnerability). Stopping the pipeline will prevent pushing vulnerable images to the container image registry.

Push stage

image:push:
  stage: push
  image:
    name: gcr.io/go-containerregistry/crane:debug
    entrypoint: [""]
  script:
    - crane auth login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - crane push build/$CI_IMAGE_TAG.tar $CI_REGISTRY_IMAGE:$CI_IMAGE_TAG
  needs:
    - image:scan
Code language: Perl (perl)

The final step authenticates and pushes the container image using crane, only if the image scanning step has been successful.

Image scanning on GitLab

Provided you have a Dockerfile in your repository and a valid Secure API token, the commit of the pipeline you just created should trigger the execution, build the image, and scan it.

You can navigate to the CI/CD -> Pipelines section of the repo to see the result of the execution:

And click into every step of the execution to get a detailed view:

The analysis results are posted to your Sysdig Secure account under Vulnerability -> Pipeline:

Success! No vulnerabilities were found, the image has been published by pushing it to the registry, and the assets are stored automatically in the cache for the next executions to save time and avoid unnecessary downloads.

This example scan used the default “Sysdig Best Practices” policy (you can see it on the logs), but you can create and customize the policies you want to check against, including not just vulnerability policies, but also image best practices.

If the scan failed because a policy failed, the workflow stops and the image is not pushed to the registry (which is a good idea) as you can see here:

There weren’t any vulnerabilities found in this example (yay!), but if we look at another application, such as https://github.com/sysdiglabs/dummy-vuln-app, we can see some were discovered:

You can filter the ones that have fixes already and/or are exploitable, focusing on the most urgent ones to fix or update:

You can not only see vulnerabilities, but also some not best practices:

Conclusions

As you can see, GitLab CI/CD is a powerful tool to automate your CI/CD pipelines directly on your GitLab repository. Now, it is easy and straightforward to include Sysdig Secure Inline Scan in your workflow, scanning images for vulnerabilities, enforcing best practices at build time, and providing several benefits over traditional image scanning within the registry:

  • Implementing image scanning in the CI/CD pipeline means that if vulnerabilities or misconfigurations are found, you prevent the image from being published at all.
  • As analysis is performed inline (locally) in the runner, the image is not sent anywhere else, including outside of the environment in which it’s built. During the analysis, only metadata information, and not the actual contents, is extracted from the image.
  • Sysdig Secure provides out-of-the-box policies to enforce and adhere to various container compliance standards (like NIST 800-190 and PCI).

Sysdig Secure Image Scanning can be integrated seamlessly with most CI/CD pipeline tools.

If you are not using Sysdig Secure for scanning your images yet, wait no longer and request a demo now!

Subscribe and get the latest updates