In this guide, we’ll cover:

1. Intro to Prometheus and its core concepts
2. How Prometheus compares to other monitoring solutions
3. How to install Prometheus
4. Monitoring a Kubernetes Service
5. Monitoring a Kubernetes cluster

After this article, you’ll be ready to dig deeper into Kubernetes monitoring. Additional reads in our blog will help you configure additional components of the Prometheus stack inside Kubernetes (Alertmanager, push gateway, grafana, external storage), setup the Prometheus operator with Custom ResourceDefinitions (to automate the Kubernetes deployment for Prometheus), and prepare for the challenges using Prometheus at scale.

Why use Prometheus for Kubernetes monitoring

Two technology shifts took place that created a need for a new monitoring framework:

• DevOps culture: Prior to the emergence of DevOps, monitoring consisted of hosts, networks, and services. Now, developers need the ability to easily integrate app and business related metrics as an organic part of the infrastructure, because they are more involved in the CI/CD pipeline and can do a lot of operations-debugging on their own. Monitoring needed to be democratized, made more accessible, and cover additional layers of the stack.
• Containers and Kubernetes: Container-based infrastructures are radically changing how we do logging, debugging, high-availability, etc., and monitoring is not an exception. Now you have a huge number of volatile software entities, services, virtual network addresses, and exposed metrics that suddenly appear or vanish. Traditional monitoring tools are not designed to handle this.

Why is Prometheus the right tool for containerized environments? These four characteristics made Prometheus the de-facto standard for Kubernetes monitoring:

• Multi-dimensional data model: The model is based on key-value pairs, similar to how Kubernetes organizes infrastructure metadata using labels. It allows for flexible and accurate time series data, powering its Prometheus query language.
• Accessible format and protocols: Exposing prometheus metrics is a pretty straightforward task. Metrics are human readable, in a self-explanatory format, and published using a standard HTTP transport. You can check that the metrics are correctly exposed by using your web browser:

• Service discovery: The Prometheus server is in charge of periodically scraping the targets so that applications and services don’t need to worry about emitting data (metrics are pulled, not pushed). These Prometheus servers have several methods to auto-discover scrape targets. Some of them can be configured to filter and match container metadata, making it an excellent fit for ephemeral Kubernetes workloads.
• Modular and high available components: Metric collection, alerting, graphical visualization, etc. are performed by different composable services. All of these services are designed to support redundancy and sharding.

How Prometheus compares to other Kubernetes monitoring tools

Prometheus released version 1.0 during 2016, so it’s a fairly recent technology. There were a wealth of tried-and-tested monitoring tools available when Prometheus first appeared. So, how does Prometheus compare with these other veteran monitoring projects?

Key-value vs dot-separated dimensions: Several engines like StatsD/Graphite use an explicit dot-separated format to express dimensions, effectively generating a new metric per label:

current_active_users.free_tier = 423
current_active_users.premium = 56

This method can become cumbersome when trying to expose highly dimensional data (containing lots of different labels per metric). Flexible, query-based aggregation becomes more difficult as well.

Imagine that you have 10 servers and want to group by error code. Using key-value, you can simply group the flat metric by {http_code="500"}. Using dot-separated dimensions, you will have a big number of independent metrics that you need to aggregate using expressions.

Event logging vs. metrics recording: InfluxDB / Kapacitor are more similar to the Prometheus stack. They use label-based dimensionality and the same data compression algorithms. Influx is, however, more suitable for event logging due to its nanosecond time resolution and ability to merge different event logs. Prometheus is more suitable for metrics collection and has a more powerful query language to inspect them.

Blackbox vs whitebox monitoring: As we mentioned before, tools like Nagios/Icinga/Sensu are suitable for host/network/service monitoring and classical sysadmin tasks. Nagios, for example, is host-based. If you want to get internal detail about the state of your micro-services (aka whitebox monitoring), Prometheus is a more appropriate tool.

The challenges of microservices and Kubernetes monitoring

There are unique challenges to monitoring a Kubernetes cluster that need to be solved in order to deploy a reliable monitoring / alerting / graphing architecture.

Monitoring containers: visibility

Containers are lightweight, mostly immutable black boxes, which can present monitoring challenges.

The Kubernetes API and the kube-state-metrics (which natively uses prometheus metrics) solve part of this problem by exposing Kubernetes internal data, such as the number of desired / running replicas in a deployment, unschedulable nodes, etc.

Prometheus is a good fit for microservices because you just need to expose a metrics port, and don’t need to add too much complexity or run additional services. Often, the service itself is already presenting a HTTP interface, and the developer just needs to add an additional path like /metrics.

In some cases, the service is not prepared to serve Prometheus metrics and you can’t modify the code to support it. In that case, you need to deploy a Prometheus exporter bundled with the service, often as a sidecar container of the same pod.

Dynamic monitoring: changing and volatile infrastructure

As we mentioned before, ephemeral entities that can start or stop reporting any time are a problem for classical, more static monitoring systems.

Prometheus has several autodiscover mechanisms to deal with this. The most relevant for this guide are:

Consul: A tool for service discovery and configuration. Consul is distributed, highly available, and extremely scalable.

Kubernetes: Kubernetes SD configurations allow retrieving scrape targets from Kubernetes’ REST API, and always stay synchronized with the cluster state.

Prometheus Operator: To automatically generate monitoring target configurations based on familiar Kubernetes label queries. We will focus on this deployment option later on.

Monitoring new layers of infrastructure: Kubernetes components

Using Kubernetes concepts like the physical host or service port become less relevant. You need to organize monitoring around different groupings like microservice performance (with different pods scattered around multiple nodes), namespace, deployment versions, etc.

Using the label-based data model of Prometheus together with the PromQL, you can easily adapt to these new scopes.

Kubernetes monitoring with Prometheus: Architecture overview

We will get into more detail later on. This diagram covers the basic entities we want to deploy in our Kubernetes cluster:

1. The Prometheus servers need as much target auto discovery as possible. There are several options to achieve this:
   • Prometheus Kubernetes SD (service discovery)
   • The Prometheus operator and its Custom Resource Definitions
   • Consul SD
   • Azure SD for Azure VM
   • GCE SD for GCP instances
   • EC2 SD for AWS VM
   • File SD
2. Apart from application metrics, we want Prometheus to collect metrics related to the Kubernetes services, nodes, and orchestration status.
   • Node exporter for the classical host-related metrics: cpu, mem, network, etc.
   • Kube-state-metrics for orchestration and cluster level metrics: deployments, pod metrics, resource reservation, etc.
   • Kubernetes control plane metrics: kubelet, etcd, dns, scheduler, etc.
3. Prometheus can configure rules to trigger alerts using PromQL. alertmanager will be in charge of managing alert notification, grouping, inhibition, etc.
4. The AlertManager component configures the receivers and gateways to deliver alert notifications.
5. Grafana can pull metrics from any number of Prometheus servers and display panels and Dashboards.

How to install Prometheus

There are different ways to install Prometheus in your host or in your Kubernetes cluster:

• As a single binary running on your hosts, which is fine for learning, testing, and developing purposes but is not appropriate for a containerized deployment.
• As a Docker container which has, in turn, several orchestration options: Raw Docker containers, Kubernetes Deployments / StatefulSets, the Helm Kubernetes package manager, Kubernetes operators, etc.

Let’s start with a more manual approach to a more automated process:

Single → Docker container → Helm chart → Prometheus operator

You can directly download and run the Prometheus binary in your host:

prometheus-2.21.0.linux-amd64$ ./prometheus
./prometheus
level=info ts=2020-09-25T10:04:24.911Z caller=main.go:310 msg="No time or size retention was set so using the default time retention" duration=15d
[…]
level=info ts=2020-09-25T10:04:24.916Z caller=main.go:673 msg="Server is ready to receive web requests."

Which may be nice to get a first impression of the Prometheus web interface (port 9090 by default).

A better option is to deploy the Prometheus server inside a container:

docker run -p 9090:9090 -v /tmp/prometheus.yml:/etc/prometheus/prometheus.yml \
       prom/prometheus

Note that you can easily adapt this Docker container into a proper Kubernetes Deployment object that will mount the configuration from a ConfigMap, expose a service, deploy multiple replicas, etc. The easiest way to install Prometheus in Kubernetes is using Helm.

The Prometheus community is maintaining a Helm chart that makes it really easy to install and configure Prometheus and the different applications that form the ecosystem.

To install Prometheus in your Kubernetes cluster with helm just run the following commands:

Add the Prometheus charts repository to your helm configuration:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add stable https://kubernetes-charts.storage.googleapis.com/
helm repo update

Install Prometheus:

# Helm 3
helm install [RELEASE_NAME] prometheus-community/prometheus
# Helm 2
helm install --name [RELEASE_NAME] prometheus-community/prometheus

After a few seconds, you should see the Prometheus pods in your cluster.

NAME                                                      READY   STATUS    RESTARTS   AGE
prometheus-kube-state-metrics-66cc6888bd-x9llw   1/1     Running   0          93d
prometheus-node-exporter-h2qx5                   1/1     Running   0          10d
prometheus-node-exporter-k6jvh                   1/1     Running   0          10d
prometheus-node-exporter-thtsr                   1/1     Running   0          10d
prometheus-server-0                              2/2     Running   0          90m

Bonus point: Helm chart deploys node-exporter, kube-state-metrics, and alertmanager along with Prometheus, so you will be able to start monitoring nodes and the cluster state right away.

A more advanced and automated option is to use the Prometheus operator. You can think of it as a meta-deployment, a deployment that manages other deployments and configures and updates them according to high-level service specifications.

How to monitor a Kubernetes service with Prometheus

Prometheus metrics are exposed by services through HTTP(S), and there are several advantages of this approach compared to other similar monitoring solutions:

• You don’t need to install a service agent, just expose a web port. Prometheus servers will regularly scrape (pull), so you don’t need to worry about pushing metrics or configuring a remote endpoint either.
• Several microservices already use HTTP for their regular functionality, and you can reuse that internal web server and just add a folder like /metrics.
• The metrics format itself is human-readable and easy to grasp. If you are the maintainer of the microservice code, you can start publishing metrics without much complexity or learning required.

Some services are designed to expose Prometheus metrics from the ground up (the Kubernetes kubelet, Traefik web proxy, Istio microservice mesh, etc.). Other services are not natively integrated but can be easily adapted using an exporter. An exporter is a service that collects service stats and “translates” them to Prometheus metrics ready to be scraped. There are examples of both in this guide.

Let’s start with the best case scenario: the microservice that you are deploying already offers a Prometheus endpoint.

Traefik is a reverse proxy designed to be tightly integrated with microservices and containers. A common use case for Traefik is as an Ingress controller or Entrypoint. This is the bridge between the Internet and the specific microservices inside your cluster.

You have several options to install Traefik and a Kubernetes-specific install guide. If you just want a simple Traefik deployment with Prometheus support up and running quickly, use the following commands:

helm repo add stable https://kubernetes-charts.storage.googleapis.com/
helm install traefik stable/traefik --set metrics.prometheus.enabled=true

Once the Traefik pods are running, you can display the service IP:

$ kubectl get svc
k get svc
NAME         TYPE            CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)                     AGE
kubernetes  ClusterIP        100.64.0.1       <none>                                                                    443/TCP                     99d
traefik    LoadBalancer      100.65.9.227 xxx.eu-west-1.elb.amazonaws.com   443:32164/TCP,80:31829/TCP  72m
traefik-prometheus ClusterIP 100.66.30.208    <none>                                                                    9100/TCP                    72m

You can check that the Prometheus metrics are being exposed in the service traefik-prometheus by just using curl from a shell in any container:

$ curl 100.66.30.208:9100/metrics
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 2.4895e-05
go_gc_duration_seconds{quantile="0.25"} 4.4988e-05
...

Now, you need to add the new target to the prometheus.yml conf file. Check it with the command:

kubectl get cm prometheus-server -o yaml

You will notice that Prometheus automatically scrapes itself:

  - job_name: 'prometheus'
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
    - targets: ['localhost:9090']

Let’s add another static endpoint.

Edit the file with the command:

kubectl edit cm prometheus-server

And add this new job:

  - job_name: 'traefik'
    static_configs:
    - targets: ['traefik-prometheus:9100]

If the service is in a different namespace, you need to use the FQDN (e.g., traefik-prometheus.[namespace].svc.cluster.local)

Of course, this is a bare-minimum configuration and the scrape config supports multiple parameters.

To name a few:

• basic_auth and bearer_token: Your endpoints may require authentication over HTTPS, using a classical login/password scheme or a bearer token in the request headers.
• kubernetes_sd_configs or consul_sd_configs: Different endpoint autodiscovery methods.
• scrape_interval, scrape_limit, scrape_timeout: Different tradeoffs between precision, resilience, and system load.

If you access the /targets URL in the Prometheus web interface, you should see the Traefik endpoint UP:

Using the main web interface, we can locate some traefik metrics (very few of them, because we don’t have any Traefik frontends or backends configured for this example) and retrieve its values:

We already have a Prometheus on Kubernetes working example.

In addition to the use of static targets in the configuration, Prometheus implements a really interesting service discovery in Kubernetes, allowing us to add targets annotating pods or services with these metadata:

annotations:
  prometheus.io/port: 9216
  prometheus.io/scrape: true

You have to indicate Prometheus to scrape the pod or service and include information of the port exposing metrics.

How to monitor Kubernetes services with Prometheus exporters

Although some services and applications are already adopting the Prometheus metrics format and provide endpoints for this purpose, many popular server applications like Nginx or PostgreSQL are much older than the Prometheus metrics / OpenMetrics popularization. This complicates getting metrics from them into a single pane of glass, since they usually have their own metrics formats and exposition methods.

If you are trying to unify your metric pipeline across many microservices and hosts using Prometheus metrics, this may be a problem.

Prometheus exporters to the rescue

To work around this hurdle, the Prometheus community is creating and maintaining a vast collection of Prometheus exporters. An exporter is a “translator” or “adapter” program that is able to collect the server native metrics (or generate its own data observing the server behavior) and re-publish them using the Prometheus metrics format and HTTP protocol transports.

These exporter small binaries can be co-located in the same pod as a sidecar of the main server that is being monitored, or isolated in their own pod or even a different infrastructure.

The exporter exposes the service metrics converted into Prometheus metrics, so you just need to scrape the exporter.

Cutting through the Prometheus exporters noise with PromCat

There are hundreds of Prometheus exporters available on the internet, and each exporter is as different as the application that they generate metrics for. In most of the cases, the exporter will need an authentication method to access the application and generate metrics. These authentications come in a wide range of forms, from plain text url connection strings to certificates or dedicated users with special permissions inside of the application. In other escenarios, it may need to mount a shared volume with the application to parse logs or files, for example. Also, the application sometimes needs some tuning or special configuration to allow the exporter to get the data and generate metrics.

Sometimes, there are more than one exporter for the same application. This can be due to different offered features, forked discontinued projects, or even that different versions of the application work with different exporters. It’s important to correctly identify the application that you want to monitor, the metrics that you need, and the proper exporter that can give you the best approach to your monitoring solution.

Sysdig has created a site called PromCat.io to reduce the amount of maintenance needed to find, validate, and configure these exporters. At PromCat.io, we curate the best exporters, provide detailed configuration examples, and provide support for our customers who want to use them. Check the up-to-date list of available Prometheus exporters and integrations.

Hands on: Monitoring redis as a Kubernetes service with Prometheus

We’ll see how to use a Prometheus exporter to monitor a Redis server that is running in your Kubernetes cluster.

You can deploy a Prometheus sidecar container along with the pod containing the Redis server by using our example deployment:

# Clone the repo if you don't have it already
git clone git@github.com:mateobur/prometheus-monitoring-guide.git
kubectl create -f prometheus-monitoring-guide/redis_prometheus_exporter.yaml

If you display the Redis pod, you will notice it has two containers inside:

# kubectl get pod redis-546f6c4c9c-lmf6z
NAME                     READY     STATUS    RESTARTS   AGE
redis-546f6c4c9c-lmf6z   2/2       Running   0          2m

Now, you just need to update the Prometheus configuration and reload like we did in the last section:

  - job_name: 'redis'
    static_configs:
      - targets: ['redis:9121']

To obtain all of the Redis service metrics:

Monitoring Kubernetes cluster with Prometheus and kube-state-metrics

In addition to monitoring the services deployed in the cluster, you also want to monitor the Kubernetes cluster itself. Three aspects of cluster monitoring to consider are:

• The Kubernetes hosts (nodes): Classic sysadmin metrics such as cpu, load, disk, memory, etc.
• Orchestration level metrics: Deployment state, resource requests, scheduling and api server latency, etc.
• Internal kube-system components: Detailed service metrics for the scheduler, controller manager, dns service, etc.

The Kubernetes internal monitoring architecture has recently experienced some changes that we will try to summarize here. For more information, you can read its design proposal.

Kubernetes monitoring components on a Prometheus stack

cAdvisor is an open source container resource usage and performance analysis agent. It is purpose-built for containers and supports Docker containers natively. In Kubernetes, cAdvisor runs as part of the Kubelet binary. So, any aggregator retrieving “node local” and Docker metrics will directly scrape the Kubelet Prometheus endpoints.

Kube-state-metrics is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects such as deployments, nodes, and pods. It is important to note that kube-state-metrics is just a metrics endpoint. Other entities need to scrape it and provide long term storage (e.g., the Prometheus server).

Metrics-server is a cluster-wide aggregator of resource usage data. The metrics server will only present the last data points and it’s not in charge of long term storage.

Thus:

• Kube-state metrics are focused on orchestration metadata: deployment, pod, replica status, etc.
• Metrics-server is focused on implementing the resource metrics API: CPU, file descriptors, memory, request latencies, etc.

Monitoring the Kubernetes nodes with Prometheus

The Kubernetes nodes or hosts need to be monitored.

We have plenty of tools to monitor a Linux host, but they are not designed to be easily run on Kubernetes.

Thus, we’ll use the Prometheus node-exporter that was created with containers in mind:

• It’s hosted by the Prometheus project itself.
• It’s the one that will be automatically deployed in our Prometheus operator examples.
• It can be deployed as a DaemonSet and will automatically scale if you add or remove nodes from your cluster.

The easiest way to install it is by using Helm:

# add repo only needed if it wasn't done before
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
# Helm 3
helm install [RELEASE_NAME] prometheus-community/prometheus-node-exporter
# Helm 2
helm install --name [RELEASE_NAME] prometheus-community/prometheus-node-exporter

Once the chart is installed and running, you can display the service that you need to scrape:

TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                     AGE
node-exporter-prometheus-node-exporter   ClusterIP   10.101.57.207    <none>        9100/TCP                                    17m

Once you add the scrape config like we did in the previous sections (If you installed Prometheus with Helm, there is no need to configuring anything as it comes out-of-the-box), you can start collecting and displaying the node metrics.

Monitoring kube-state-metrics with Prometheus

Deploying and monitoring the kube-state-metrics just requires a few steps.

Again, you can deploy it directly using the commands below, or with a Helm chart. If you installed Prometheus with Helm, kube-state-metrics will already be installed and you can skip this step.

git clone https://github.com/kubernetes/kube-state-metrics.git
kubectl apply -f examples/standard
...
# kubectl get svc -n kube-system
NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
kube-dns             ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP       13h
kube-state-metrics   ClusterIP   10.102.12.190    <none>        8080/TCP,8081/TCP   1h

You just need to scrape that service (port 8080) in the Prometheus config. Remember to use the FQDN this time:

  - job_name: 'kube-state-metrics'
    static_configs:
    - targets: ['kube-state-metrics.kube-system.svc.cluster.local:8080']

Monitoring Kubernetes control plane with Prometheus

The control plane is the brain and heart of Kubernetes. All of its components are important to the proper working and efficiency of the cluster. Monitoring the Kubernetes control plane is just as important as monitoring the status of the nodes or the applications running inside. It may be even more important, because an issue with the control plane will affect all of the applications and cause potential outages.

There are several Kubernetes components that can expose internal performance metrics using Prometheus. Check these other articles for detailed instructions, as well as recommended metrics and alerts:

• How to monitor Kubernetes apiserver
• How to monitor kubelet
• How to monitor etcd
• How to monitor controller-manager
• How to monitor kube-proxy
• How to monitor kube-dns

Monitoring them is quite similar to monitoring any other Prometheus endpoint with two particularities:

• The network interfaces these processes listen to, and the http scheme and security (HTTP, HTTPS, RBAC), depend on your deployment method and configuration templates. Frequently, these services are only listening at localhost in the hosting node, making them difficult to reach from the Prometheus pods.
• These components may not have a Kubernetes service pointing to the pods, but you can always create it.

Depending on your deployment method and configuration, the Kubernetes services may be listening on the local host only.

To make the next example easier and focused, we’ll use Minikube. Minikube lets you spawn a local single-node Kubernetes virtual machine in minutes.

This will work as well on your hosted cluster, GKE, AWS, etc., but you will need to reach the service port by either modifying the configuration and restarting the services, or providing additional network routes.

Installing Minikube only requires a few commands. First, install the binary, then create a cluster that exposes the kube-scheduler service on all interfaces:

minikube start --memory=4096 --bootstrapper=kubeadm --extra-config=kubelet.authentication-token-webhook=true --extra-config=kubelet.authorization-mode=Webhook --extra-config=scheduler.address=0.0.0.0 --extra-config=controller-manager.address=0.0.0.0

Then, we can create a service that will point to the kube-scheduler pod:

kind: Service
apiVersion: v1
metadata:
  name: scheduler-service
  namespace: kube-system
spec:
  selector:
    component: kube-scheduler
  ports:
  - name: scheduler
    protocol: TCP
    port: 10251
    targetPort: 10251

Now you will be able to scrape the endpoint: scheduler-service.kube-system.svc.cluster.local:10251

Prometheus at scale

Monitoring with Prometheus is easy at first. You can have metrics and alerts in several services in no time. The problems start when you have to manage several clusters with hundreds of microservices running inside, and different development teams deploying at the same time.

Global visibility, high availability, access control (RBAC), and security are requirements that need to add additional components to Prometheus, making the monitoring stack much more complex.

There are unique challenges using Prometheus at scale, and there are a good number of open source tools like Cortex and Thanos that are closing the gap and adding new features.

There is also an ecosystem of vendors, like Sysdig, offering enterprise solutions built around Prometheus.

What’s next?

We have covered basic prometheus installation and configuration. But now it’s time to start building a full monitoring stack, with visualization and alerts.

We suggest you continue learning about the additional components that are typically deployed together with the Prometheus service. We will start using the PromQL language to aggregate metrics, fire alerts, and generate visualization dashboards.

Further reads in our blog will help you set up the Prometheus operator with Custom ResourceDefinitions (to automate the Kubernetes deployment for Prometheus), and prepare for the challenges using Prometheus at scale.

You may also find our Kubernetes monitoring guide interesting, which compiles all of this knowledge in PDF format.

Want to put all of this PromQL, and the PromCat integrations, to the test? Sysdig Monitor is fully compatible with Prometheus and only takes a few minutes to set up. Start your free trial today!

The post Kubernetes monitoring with Prometheus, the ultimate guide appeared first on Sysdig.

Implementing a network policy is challenging, as developers and ops need to work together to define proper rules. However, the best approach is to adopt a zero trust framework for network security using Kubernetes native controls (network policies).

Learn how Sysdig Secure is closing this gap with its latest Sysdig Network Policy feature that provides Kubernetes-native network security, allowing both teams to author the best possible network policy, without paying the price of learning yet another policy language. It also helps meet compliance requirements (NIST, PCI, etc) that require network segmentation.

Network Policy Needs Context

Kubernetes communications without metadata enrichment are scribbled.

Before you can even start thinking about security and network policies, you first need to have deep visibility into how microservices are communicating with each other.

In a Kubernetes-world, pods are short-lived, they jump between hosts, have ephemeral IP addresses, scale up and down. All that is awesome and gives you the flexibility and reactiveness that you love. But it also means that if you look at the physical communication layer at L3/L4 (just IPs and ports, it looks like this:

In other words, you have all the connection information, but it is not correctly aggregated and segmented. As you can expect, trying to configure classic firewall rules is not going to make it. Yes, you can start grouping the different containers according to attributes like image name, image tags, container names… But this is laborious, error prone and remember all that information is dynamic and constantly changing, it will always be an uphill battle.

Why reinvent the wheel when you already have all the metadata you need to reason about network communications and security at the Kubernetes level? The Kubernetes API contains all the up-to-date information about namespaces, services, deployments, etc. It also provides you a tool to create policies based on the labels assigned to those entities (KNPs).

Creating a Kubernetes network policy

Your developers team met with your ops team to create the ultimate network policy for one of your apps.

After one hour of meeting, this is how developers defined the app:

• The “example-java-app” queries two mongoDB databases, one local to the cluster another external, it also needs a redis cache.
• It receives requests from an “example-java-client” app.

And this is the network policy they came up with.

Let’s start the policy with the usual metadata. This is a NetworkPolicy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: generated-network-policy
  namespace: example-java-app

Then the network rules. Starting with an ingress rule so our example-java-app can accept requests from the client app on port 8080:

spec:
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              app: raw
              chart: raw-0.2.3
              heritage: Helm
              release: namespaces
          podSelector:
            matchLabels:
              app.kubernetes.io/instance: example-java-app
              app.kubernetes.io/name: example-java-app-jclient
      ports:
        - port: 8080
          protocol: TCP

Then egress rules so our app can connect to our databases: mongodb, and redis.

  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              app: raw
              chart: raw-0.2.3
              heritage: Helm
              release: namespaces
          podSelector:
            matchLabels:
              app.kubernetes.io/instance: example-java-app
              app.kubernetes.io/name: example-java-app-mongo
      ports:
        - port: 27017
          protocol: TCP
    - to:
        - namespaceSelector:
            matchLabels:
              app: raw
              chart: raw-0.2.3
              heritage: Helm
              release: namespaces
          podSelector:
            matchLabels:
              app.kubernetes.io/instance: example-java-app
              app.kubernetes.io/name: example-java-app-redis
      ports:
        - port: 6379
          protocol: TCP

You finish by specifying this will apply to the pod named example-java-app-javaapp:

  podSelector:
    matchLabels:
      app.kubernetes.io/instance: example-java-app
      app.kubernetes.io/name: example-java-app-javaapp
  policyTypes:
    - Ingress
    - Egress

Wow, this is a long YAML for such a simple app. A network policy for a regular app can expand up to thousands of lines.

And this is the first caveat of Kubernetes network policies, you need to learn yet another domain specific language to create them. Not every developer will  be able to or want to,  and the ops team doesn’t know enough about your apps to create them. They need to work together.

Let’s deploy this network policy in production!

After applying the policy, the app stops working 😱

And this is another caveat of the process of creating Kubernetes network policies. Troubleshooting needs several stakeholders involved, which slows down the process. It is really easy to forget about some services or common KNP pitfalls when creating a network policy by hand.

What did you miss?

• You forgot a connection to an external service, ip 192.229.173.207.
• There was a deprecated connection to a Cassandra database, not fully removed.
• You forgot to allow DNS, so although services could talk to each other, they cannot resolve their service names to IP addresses.

Of course, the teams didn’t find them all at once. It was a trial and error process that spanned a few hours. You fixed it by completely removing the Cassandra dependencies, and adding these egress rules:

Rule to allow traffic to an external IP:

[...]
spec:
[...]
  egress:
  [...]
    - to:
        - ipBlock:
            cidr: 192.229.173.207/32
            except: []
[...]

Rule to allow traffic for DNS resolution:

[...]
spec:
[...]
  egress:
  [...]
    - to:
        - namespaceSelector: {}
      ports:
        - port: 53
          protocol: UDP
[...]

And finally, after applying this updated policy, everything is working as expected. And our team rest assured that our app network is secured.

The challenge of creating Kubernetes network policies

Let’s briefly stop to analyze why this process didn’t make people happy.

Kubernetes network policies are not trivial

Ok, it’s easy to start using them. KNPs come out of the box in Kubernetes, assuming you deployed a CNI that supports them, like Calico. But still, they have many implementation details and caveats:

• They are allow-only, everything that is not explicitly allowed is forbidden.
• They are based on the entities (pods, namespaces) labels, not on their names, which can be a little misleading at first.
• You need to enable DNS traffic explicitly, at least inside the cluster.
• Etc…

They are another policy language to learn and master.

The information needed is spread across teams

Developers typically have a very accurate picture of what the application should be doing at the functional level: Communicating with the database, pulling data from an external feed, etc. But are not the ones applying and managing the KNPs in production clusters.

Ops teams live and breath Kubernetes, but they have a more limited visibility on how the application behaves internally, and the network connections it actually requires to function properly.

The reality is that you need both stakeholders’ input to generate the best possible network policy for your microservice.

KNPs’ learning curve and the back and forth of information between teams with different scopes have traditionally been a pain point. Some organizations have settled for a very broad policy (i.e. microservices can communicate with each other if they are in the same namespace), which is better than nothing and simple enough to be followed by everyone, but definitely not the most accurate enforcement.

Enter Sysdig Kubernetes network policies

What if you could just describe what your application is doing, in your own language, and then somebody will translate this for you to KNPs (or whatever other policy language you use in the future)?

Let’s see how Sysdig can help you create accurate network policies, without the pain.

Today, Sysdig announces a new Kubernetes network policies feature, which delivers:

• Out-of-the-box visibility into all network traffic between apps and services, with a visual topology map to help you identify communications.
• Baseline network policy that you can directly refine and modify to match your desired declarative state.
• Automated KNPs generation based on the topology baseline + user defined adjustments.

Hands on!

Let’s see how you can use the new Kubernetes network policies feature in Sysdig to create an accurate policy for our example-java-app in a few minutes.

You can create a network policy by jumping into the Sysdig interface -> Policies -> Network Security policies

Gathering information to create a Kubernetes network policy

First, select the set of pods that you want to create a policy for.

Starting with by the cluster, and the namespace where the app lives:

Then select the Pod Owner, or the group of pods, you want to apply the policy. You can group Pods by Service, Deployment, StatefulSet, DaemonSet, or Job.

In this case let’s look to our infrastructure from a Deployment perspective:

Three hours of data will be enough for Sysdig to understand the network connections from example-java-app.

You don’t need to wait for Sysdig to actually collect network data. As the Sysdig agent has been deployed in this cluster for a while, Sysdig can use existing data from the activity audit.

Right after you select these parameters, Sysdig shows the observed network topology map for our app deployment:

There are some interesting keys here.

This map shows all the current communications (connectors and ports). Including those that you missed in our initial approach, like the cassandra database, and the external ip.

Note that the communications are shown at the Kubernetes metadata level. You can see the “client” application pushing requests to a service, then they are forwarded to deployment pods.

It proposes a security policy, color-coded. Unresolved IPs, the entities that cannot be matched against a Kubernetes entity, are excluded by default and displayed in red.

It is clean, simple and contains all the relevant information you need.

Tweaking a Kubernetes network policy

Now, as a developer with knowledge of the application, there are a few details you’ll want to tweak:

• I want to allow that specific external IP.
• There are still connections to the cassandra service that you don’t want to allow moving forward.

Let’s do so from the Ingress and Egress tabs on the interface.

The Ingress / Egress tables expand and detail the information found in the Topology map:

The extra information does help you identify the individual communications. For example: “Which process is initiating the egress connection?”

They are actionable:

• You can cherry-pick the communications that you want to allow from the table.
• You can look at the unresolved IPs and decide if you want to allow them moving forward.

Let’s add an IP/mask combination to allow the external IP, as it belongs to a trusted service.

Let’s uncheck the row for the cassandra deployment, as you no longer need that communication.

Sysdig will automatically detect that our IP belongs to a network that is external to the cluster, and flags it as such.

The network topology map will be automatically updated to reflect the changes:

Notice the connection to the external ip is no longer red, but the connection to cassandra is.

Generating a Kubernetes network policy

Now that our policy looks exactly how you want it to look, you can generate the Kubernetes Network Policy YAML.

It will be generated on the fly just clicking on the “Generated Policy” tab:

Now I can attach this policy artifact together with my application for the DevOps team.

Applying the Kubernetes network policy

As you can see in the workflow above, Sysdig is helping you generate the best policy by aggregating the observed network behavior and the user adjustments.

But you are not actually enforcing this policy.

Let’s leave that to the Kubernetes control plane and CNI plugin. And this has some advantages:

• It’s a native approach that leverages out-of-the-box Kubernetes capabilities.
• You avoid directly tampering with the network communications, host iptables, and the TCP/IP stack.
• It’s portable. You can apply this policy in all your identical applications / namespaces, it will work on most Kubernetes flavors like OpenShift and Rancher.

What is so exciting about this feature?

First, you didn’t have to understand the low level detail of how KNPs work. If you are able to describe your application network behavior using a visual flow, Sysdig will do the translation for you.

The topology map provided network visibility into all communication across a service / app / namespace / tag. So you won’t forget to include any service.

You also don’t start from scratch. By baselining your network policy you only perform a 10% lift, which saves you time. Then Sysdig auto generates the YAML.

Devops and security teams only need to sign off and check the policy.

Finally, it leverages Kubernetes native controls, decoupling policy definition and enforcement. As such, it is not intrusive, and it doesn’t imply a performance hit. Also, if you lose connection to Sysdig your policies are still on and your network communications will stay up.

Conclusion

To address the Kubernetes network security requirements, Sysdig Secure just introduced its Sysdig Network Policy feature.

Using this feature you will:

• Get automatic visibility for your microservices and their communications, using the Kubernetes metadata to abstract away all the physical-layer noise.
• Apply least-privilege microsegmentation for your services
• Automatically generate the KNPs that derives from your input, no previous KNP expertise required.

You can check this feature today, start a free trial!

The post Kubernetes network policies with Sysdig appeared first on Sysdig.

What is the Atlassian Bamboo CI/CD platform?

Atlassian Bamboo is a continuous integration and delivery server integrated with Atlassian software development and collaboration platform. Some of the features that distinguish Bamboo from similar CI/CD tools are its native integration with other Atlassian products (like Jira project management and issue tracker), improved support for Git workflows (branching and merging), and flexible scalability of worker nodes using ephemeral Amazon EC2 virtual machines. Inline image scanning builds #container #security into your #CICD pipeline while keeping images private. Learn how, with Sysdig secure and @Atlassian Bamboo. Click to tweet

Bamboo organizes the build pipeline with a hierarchy of Projects, Plans, Stages, Jobs and Tasks.

A project is a “namespace” containing different plans. The plan is composed of multiple stages that will be executed sequentially. Each stage is in turn composed of several jobs that can run in parallel; these jobs are the basic unit of work scheduled in the worker nodes. Finally, each job is composed of a series of tasks that will be executed sequentially in the scheduled worker node.

To summarize:

• The plan is the high level sequence of actions (stages) that you want to execute.
• The job is the basic schedulable unit of work that gets assigned to a worker. N jobs get executed in N workers and thus, can be parallelized.
• A job itself is a sequence of tasks, like executing a script or running a Docker container.

Container security in your CI/CD pipeline: shift left your security and fail fast

Like most things in IT, the earlier you detect container security issues, the easier they are to fix without further consequences.

Embedding container security in your build pipeline is a best practice for several reasons:

• The vulnerabilities will never reach your production clusters, or even worse, a client environment.
• You can adopt a secure-by-default approach when you know any image available in your Docker container registry has already passed all the security policies you have defined for your organization, as opposed to manually checking container and Kubernetes compliance after-the-fact.
• The original container builder will be (almost) instantly informed, when the developer still has all the context. The issue will be substantially easier to fix this way than if found by any other person months later…

Sysdig Secure offers a full featured container image scanning service, among many other container security features like run-time threat detection, forensics, compliance and auditing. Let’s see how we can make Sysdig Secure image scanning in Kubernetes work together with Atlassian Bamboo.

Bamboo CI/CD pipeline image scanning with Sysdig Secure

Following a practical example, we are going to demonstrate how to integrate these two platforms with a very straightforward process.

But first, why inline scanning? Because the full analysis will be done in the pipeline itself, no information of the contents inside the image nor the image itself will be sent outside. This will guarantee the privacy of the contents and speed-up the scanning process since it can be done in parallel.

Ok, let’s start; first, we will create a new Project containing a Plan, which essentially means naming them, writing the corresponding description, configuring access credentials for users, etc:

Next, we will create an environment for the plan.

First, we need to configure the repository that is going to contain the Dockerfile together with other build variables and artifacts:

You might configure other environment variables you need from the Variables tab.

From the Global Bamboo settings (top-right gear icon, linked repositories), you can configure any credentials you require to access your source repository, like GitHub and your container image registry, DockerHub, for example.

Next, we will configure this plan to trigger automatically whenever there is a change in the repository:

Configuring the CI/CD pipeline

We are now ready to create a Stage containing the different tasks:

• Source code checkout: Cloning the supporting repository that contains the Dockerfile and other build variables.

• Build Image: We will select Build a Docker image in Command, and we will configure a staging image registry where we can upload the image.

• Pull Image Scan: We will pull the latest version of the secure-inline-scan image that will do the scan for us.

• Scan Image: Before we even push the image into the registry, we have to make sure the image won’t compromise the security of our infrastructure.

We will use the previous downloaded image to scan the one we built with the following command:

/bin/inline_scan analyze -k ${bamboo.SYSDIG_SECURE_TOKEN_SECRET} -s ${bamboo.SYSDIG_SECURE_WEBHOOK} sysdiglabs/dummy-vuln-app:latest

Since this image will need to have access to the built image, we will need to give it access to the Docker socket:

Notice that the image won’t be sent to Sysdig Secure, just the report for further analysis. This will ensure the privacy of the contents in your image and Sysdig Secure won’t act as a bottleneck if you scan many images.

If the scanning finds any vulnerabilities in the image (including custom policies that you may have defined, like specific checks on the Dockerfile, for example not running the process as root, or whitelist/blacklist certains packages or libraries like not installing SSH), the entire plan will fail and the last step will never get executed.

• Push image: This is the final step of the pipeline, pushing the image to the registry if everything was correct.

Uh-oh! It seems that Sysdig Secure container image scanner found something wrong:

Within Sysdig Secure you can see the full report, including the policy evaluation and why it failed:

It seems that our image is exposing port 22. It also contains vulnerabilities in the operating system and the python libraries we are using:

We can select Operating System and see the vulnerabilities that affect the base image:

Or select Non-operating System and see the ones that affect the libraries we are using:

Now we know what happened, why the pipeline failed, and how to fix it.

Conclusions

Integrating your security policy early in your CI/CD pipeline will prevent known software vulnerabilities to be deployed in our image registries, or even in production, and you will enforce best security practices within your build pipeline before those containers are ever run.

Thanks to the native Docker compatibility in Atlassian Bamboo and the Sysdig Secure container image scanning API, making them work together is a breeze. Find any software vulnerabilities, check for container security best practices and Dockerfile contents, whitelist or blacklist specific packages or 3rd party libraries installed manually (like Java JAR/WAR files, or package managers like npm, pip or gem) with Sysdig Secure. Fail fast, inform the container author right away to address it quickly, and create a secure-by-default container security policy.

The post Integrating Sysdig Secure with Atlassian Bamboo CI/CD. appeared first on Sysdig.

Sysdig provides a security for Cloud Run for Anthos. With Sysdig Secure you can embed security and validate compliance across the serverless lifecycle. The Sysdig platform is open by design, with the scale, performance and usability enterprises demand.

Sysdig announced its support for Google Cloud’s Anthos earlier this year. Anthos has now integrated the Cloud Run serverless technology as a new native component, which is also fully supported by Sysdig.

Although we are going to use Anthos to illustrate the examples and use cases in this post, the Cloud Run serverless platform can also be deployed to a managed Google Cloud Run environment, Google Kubernetes Engine (GKE), GKE on-prem and multi-cloud environments. Cloud Run is based on Knative, ensuring that your applications remain portable and compliant to open standards.

Let’s start with a brief introduction to the platform:

Google Cloud Anthos

Anthos is an application management platform designed for hybrid cloud environments.

Anthos bridges the gap between the managed cloud and on-prem deployments providing a unified control plane and a consistent development and operations experience across physical servers, on-prem VMs, Compute Engine VMs, etc.

Built on top of ubiquitous cloud-native technologies like Kubernetes, Istio, and Knative, Anthos will allow you to seamlessly run your container workloads wherever you need them to, simplifying the migration, modernization, and security controls for your applications.

Google Cloud Run

Cloud Run is built around the idea of combining the simplicity and abstraction of serverless computing with the flexibility of containerized workloads, in one single solution.

Using Cloud Run, you will define your containers in the usual way (i.e. writing a Dockerfile), but their infrastructure and lifecycle management will be abstracted away from you, offering a native serverless experience. Cloud Run will also automatically manage your service scale, up and down from zero depending on traffic, almost instantaneously.

Using strict serverless frameworks you are forced to follow certain constraints, having to choose from a predefined set of supported languages, libraries and code frameworks. Using a standard containerized solution is more flexible in this regard, but comes with an additional operations investment: provisioning, configuring, and managing servers. Cloud Run gives you the best of both worlds, offering the simplicity of a serverless platform without limiting the way you develop your services code.

You can build and deploy your Google Cloud Run applications over any compatible platform in three simple steps:

• Define: Describe the container writing its Dockerfile, you get to choose the base operating system, frameworks and libraries, listening ports, etc.
• Publish: Using the gcloud command line tool, the published container image is stored in the Google Container Registry associated with your GCP project.
• Deploy: Executing the run deploy subcommand for our image, together with the service name and region. On successful deployment, the command line will automatically display your service’s URL. Your service is up and ready to go!

Before we dive into more specific examples, there are several technical considerations that you need to keep in mind when deploying Cloud Run services:

• Cloud Run containers are stateless, this allows the platform to treat them as serverless functions.
• You containers are required to accept HTTP requests in the port defined on the PORT environment variable.
• Even when you are running the same container image, different configurations will create container instances with different revision IDs. This unique identifier is extremely useful to determine which workload in particular is behaving in an unexpected way, when was it launched and the YAML containing the complete definition. We will leverage that feature in the examples below to correlate executions and instances during the security forensics phase.

Serverless security with Cloud Run and Sysdig

Containers are ephemeral by nature, Cloud Run containers doubly so, since they are managed as a serverless function and scaled up and down on demand. Based on our 2019 container usage report, 52% of containers live 5 minutes or less.

Does this mean that Cloud Run security needs to be relaxed, or that security controls will become a burden for your lightning-fast service deployment pipeline? No, you just need to use the tools that are tailored for this workflow.

Sysdig Secure embeds security and compliance across all stages of the Cloud Run serverless lifecycle:

• Sysdig instrumentation will directly communicate with the host kernel, from the point of view of the pods/containers it is completely transparent.
• This single agent per host will provide you visibility, monitoring and security. A single platform that will correlate monitoring and security events, showing you the big picture.
• Sysdig natively understands container-native and Kubernetes metadata, allowing you to instantly organize and segment your data by namespace, container id, service id, etc.

Let’s showcase three common security use cases to get a taste of the possibilities of Cloud Run and Sysdig running together:

• Advanced image scanning for Cloud Run containers
• Runtime security for serverless Cloud Run workloads
• Forensics and incident response in a serverless context

Advanced image scanning for Cloud Run containers

As we mentioned earlier, after defining and building the container image that will be used for Cloud Run, you will need to publish it in your project’s Google Cloud Registry.

This step is pretty straightforward, follow these instructions to configure your Docker access credentials, tag your image according to your Google Cloud project ID and just:

docker push gcr.io/<your_project_id>/hello:latest

Now, we need to configure the Sysdig Secure backend to access the same repository that you are using for your Cloud Run project.

From the Sysdig Secure interface access the Image Scanning section (on the left) and click on registry credentials:

From this interface, you can add new container registries. Let’s go ahead and configure the GCR repository that we will be using for Cloud Run.

In the form, you will need to set the repository path, pick Google Container Registry as the registry type and obtain the JSON auth key associated with your account. You can, optionally, download an image from that repository to test connectivity and credentials.

Save the form, now your Cloud Run images can be pulled and analyzed by Sysdig Secure. If you look at the report from that particular image:

You will be able to browse:

• Base distro/version for this image
• Vulnerabilities found at the OS level
• Vulnerabilities found at Non-OS level (Python, npm, ruby gems, etc)
• Policy evaluation criteria and final decision for this image (pass|fail)
• Inventory for the content and files in this image, etc.

Detecting CVEs in your images is just the first step, you can configure your own customized policies or run queries to report on CVEs or package in run-time or static environments.

Runtime security for serverless Cloud Run workloads

Image scanning and vulnerability reporting is an essential part of your cloud-native security, but security teams still need to detect and respond to attacks at runtime:

• A new 0-day vulnerability is released in the wild
• Malware managed to go through the scanning phase undetected
• You containers or hosts are accessed interactively to exploit a privilege escalation
• Your application binary is secure, but the deployment configuration is not, etc

There are many security scenarios where you will require container-specific runtime security to adequately respond to these threats.

Google Cloud Run services are simple: You can immediately figure out which binaries they need to run, which ports they need to open, which files they need to access, etc. Using the Sysdig runtime behavioural language you can adopt a least privilege policy for your Cloud Run services, everything that is now allowed will be automatically flagged.

Sysdig will analyze live the runtime behaviour of every container and the host, including serverless applications supported by Cloud Run containers. All this information is fed to the runtime rule engine, which will trigger an alert if any rule violation is detected.

Let’s illustrate this with a simple example. We just launched our “hello” service using Google Cloud Run for Anthos:

As you can see in the image above, the same service has different revision IDs, whenever you change the configuration and re-launch the service, it will be assigned a new unique ID.

Using Sysdig, you will have full visibility over the containers that support your Cloud Run services, out of the box. Accessing either the Monitor or Secure interfaces of the Sysdig platform, you will be able to see the existing Cloud Run containers organized hierarchically using their Kubernetes metadata (namespace, deployment, annotations, labels).

The Sysdig agent will capture the Kubernetes labels and tags for every running container. For the Cloud Run context, you can use these tags for two immediately relevant use cases:

• Classifying and telling the Cloud Run managed containers apart from the other “regular” containers in your cluster
  • This will allow you to define different visualizations, security alerts, metric thresholds, etc depending on the “type” of container that you are scoping to.
• Retrieve the Revision ID for any current or past container that was running on your cluster and correlate this information with the security alerts or service performance events.

We know that our simple “hello” container is just executing a process:

root         1  0.0  0.1 110760 11976 ?        Ssl  11:18   0:10 /hello

Let’s create a whitelisting rule that will alert you if any other binary is executed inside that container.

You can access the runtime rule library in Sysdig Secure and create a new runtime rule:

You will just create a new rule that triggers whenever the syscall stream detects any binary that is not in the whitelist:

Now, let’s scope and make this rule actionable creating a runtime policy:

We are scoping this policy to containers where the tag revision contains the string hello, it will work for all the different revision IDs for the same serverless service, but it won’t affect the other regular containers living in our cluster.

Let’s trigger this policy, for example executing ls inside the container, you will soon receive an event like this:

If you check the full details for the event, you will be able to display:

• Complete Kubernetes scope (host, namespace, pod name, revision ID)
• Commands audit information (PID, PPID, shell id, UID, cwd)
• Container id, full container name in Kubernetes, Image SHA256 digest

For the sake of simplicity we just demonstrated a basic runtime rule in this example. Using the Falco runtime language you can describe any advanced trigger condition that you need to enforce in your scenario, including syscalls, container labels, kubernetes attributes, lists, macros with subconditions and so on:

Forensics and incident response in a serverless context

Google Cloud Run is a serverless, autoscaled-on-demand environment. This means that the affected workload will probably be long gone when the security incident comes to your attention. How do you perform incident response and forensics in such scenario? It’s like the CSI series, but without a body.

Again, it’s just a matter of using the right tool for the job. Going back to the policy definition that we displayed before there are several automated actions that you can take when a policy is triggered:

• You will always notify the event in the events stream, directly stop the container or pause it for further investigation.
• You have several notification channels to push the event information (email, webhook, slack, pagerduty, etc).
• You can also create a full activity capture. Note that you can capture after, but also before the security event happened.

This is an important detail for forensics, you want to be able to reconstruct the sequence of events prior to the security trigger. Many attacks are preceded by recognizance and information discovery steps.

For this example, we are going to use the Unexpected outbound connection destination. You know that your Cloud Run service needs to contact several Google Cloud APIs, and you can whitelist those, but if it tries to contact any other destination that is suspicious (hijacked to mine bitcoins? Trying to connect to the regular pods in your cluster? Trying to contact the K8S API?). This time we are going to configure a Sysdig capture associated with the policy event.

Now, we are going to trigger the policy. To make things more realistic, after triggering the security policy, we are going to deploy a new service revision in our Google Cloud console:

The old service revision doesn’t have any incoming traffic, so it was automatically scaled down to 0.

But the alert was triggered for the old revision:

You can correlate the information present in Sysdig Secure and the Google Cloud Run console to discover:

• The specific service revision ID that triggered the alert, when did it trigger, what runtime rule in particular was violated.
• Who launched that revision, when, and which configuration was being used at that time.

And that’s just the first steps in your forensics investigation, attached to this policy event you will find a capture with every system call that was executed during the security incident:

There seems to be some interactive activity that matches the exact moment that the alert was triggered, but also before. Let’s take a closer look at the executed interactive commands:

Extremely suspicious activity, the last command (curl) was the one actually triggering the policy. From this interface we can drill down to discover the fine-grained details of any action performed during the attack: connections open, new processes spawned, listening ports, files modified and even the content that was modified inside those files:

Conclusions

The Google Cloud Run platform sits in the sweet spot between serverless platforms and containers. Here at Sysdig, we are happy to expand our Google Cloud Anthos support and provide support Cloud Run for Anthos, for both the Monitor and Secure pillars of the platform.

The Sysdig platform brings many advantages to the Google Cloud Run workflow:

• Transparent instrumentation, that will keep your Google Cloud services light, simple, and adhering to the serverless principles.
• Native support for private Google container registries, enabling image scanning, custom image checks, and vulnerability reporting directly from the interface.
• Cloud-native and fine-grained forensics, specially tailored to deal with ephemeral and stateless workloads.

The post Securing Google Cloud Run serverless workloads appeared first on Sysdig.

• Runtime policies can import any number of security rules and auto-generated image profiles. You can scope the security policy using container, cloud and Kubernetes metadata.
• Tighter Falco integration, directly from the web UI. You will be able to define a new trigger condition or append to the list of forbidden external IPs just clicking on the rule.
• A more structured way to group, classify and lookup rules, following the standard Cloud native procedure: tags and labels.

• **Where this rule comes from **(Published By). The security team can instantly recognize whether a rule came from a specific Sysdig update, from a custom rules file created within the organization or from an external rules source (like the Falco community rules).
• When was the last time it was updated (Last Updated). You can use this information to audit your rules or if you schedule periodic updates, to confirm when last happened.
• Rule tags: An effective method for organizing your rules. You can use these tags to describe the targeted entity (host, k8s, process), the compliance standard it belongs to (MITRE, PCI, CIS Kubernetes) or any other criteria you want to use to annotate your rules.

• Falco macro conditions
• List elements

1. Click on the rule once to display a side panel containing the Falco rule syntax. This rule uses macros and lists to define desired trigger conditions and exceptions.
2. These macros and lists are highlighted, signaling that you can click on them to pop-up an inline editing dialog. Through this form, you can add your container image name to the user_privileged_containers macro.

Creating a new runtime security rule

• Container images
• File system operations (read/write directories, read directories)
• TCP/UDP ports, or inbound / outbound network connections
• Spawned processes
• System calls

The new Sysdig Secure policy editor

• Import one or several rules that will trigger this policy, directly from the Rules Library or creating new rules from this interface.
• **Scope the policy **using labels and tags. Enforcing this rule over a Kubernetes namespace or the hosts in a specific availability zone, to give you an example.
• Configure automated remediation actions: stop the container, create a capture to enable advanced container forensics or forward the event to a SIEM platform.

Wrapping up

• Falco rules, macros and lists, tightly integrated with the interface.
• Out of the box runtime compliance standards, tagged using the different sections of the standard.
• Multiple runtime rules and image profiles per runtime policy.

The post Introducing the new Sysdig Secure policy editor appeared first on Sysdig.

Why you may need custom metrics for your Kubernetes autoscaler?

If the metrics-server plugin is installed in your cluster, you will be able to see the CPU and memory values for your cluster nodes or any of the pods. These metrics are useful for internal cluster sizing, you probably want to configure your Kubernetes autoscaler using a wider set of metrics:

• Service latency -> net.http.request.time
• I/O load -> file.iops.total
• Memory usage -> memory.used.percent

You need a metrics provider that is able to provide detailed performance information, aggregated using Kubernetes metadata (deployments, services, pod). The extension API server implementation in this post uses Sysdig Monitor.

Before we start deploying our custom Kubernetes autoscaler, let’s go over the HPA basics.

Kubernetes autoscaler (HPA) basic concepts

We can start with a simple diagram:

As you can see above, the HPA object will interact with a pod controller like a Deployment or ReplicaSet. It will update this object to configure the “desired” number of pods given the current metric readings and thresholds.

The pod controller, a Deployment for instance, will then terminate or create new pods as part of its reconciliation loop to reach the desired state.

The basic parameters that will you need for any HPA are:

• Scale target: the controller that this HPA will interact with
  • minReplicas: minimum number of pods, the HPA cannot go below this value
  • maxRepicas: maximum number of pods, the HPA cannot go above this value
• Target metric(s): metric (or metrics) used to evaluate current load and take scaling decisions
  • targetValue: threshold value for the metric. If the metric readings are above this value, and (currentReplicas < maxReplicas), HPA will scale up.

You can create a Kubernetes HPA in just one line:

$ kubectl autoscale deployment shell --min=2 --max=10 --cpu-percent=10
horizontalpodautoscaler.autoscaling/shell autoscaled

If you generate high CPU loads in these pods, the HPA will scale up the desired number of replicas:

23s         Normal    SuccessfulRescale   HorizontalPodAutoscaler   New size: 4; reason: cpu resource utilization (percentage of request) above target

It will also scale down again when the CPU burst is over. Pretty neat, right? There are many other details covering the HPA algorithm and advanced configuration details in the Kubernetes official documentation.

Like many other things in Kubernetes, the set of metrics available to the HPAs can be expanded implementing an API extension. Let’s see how this is done.

Kubernetes custom metrics API

The Kubernetes HPA is able to retrieve metrics from several APIs out of the box: metrics.k8s.io, custom.metrics.k8s.io (the one that we will use in this post), and external.metrics.k8s.io.

To register custom metrics and update their values, you need to:

• Enable the Kubernetes aggregation layer
• Register a new APIService object that will bind the new API path to the Kubernetes service implementing it
• The actual service implementation (a pod living inside a Kubernetes namespace for this example) that responds to the HPA requests and retrieves the metric values from the external provider

If you are using a recent Kubernetes version (1.11+), the API aggregation layer is probably enabled and configured out of the box, so you can skip this step. You can check the relevant API server parameters describing the kube-apiserver pod living in your kube-system namespace:

$ kubectl describe pod kube-apiserver -n kube-system
...
    Command:
      kube-apiserver
...
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
      --requestheader-allowed-names=front-proxy-client
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
      --requestheader-extra-headers-prefix=X-Remote-Extra-
      --requestheader-group-headers=X-Remote-Group
      --requestheader-username-headers=X-Remote-User
...

In case your API server doesn’t have these flags, the Kubernetes documentation has an article explaining how to configure them.

These parameters enable the kube-aggregator, a controller in charge of two tasks:

• Discovering and registering APIService objects, creating a link between the newly registered API path and the Kubernetes service implementing it.
• Acting as the front-proxy / forwarder for these requests.

This is how a basic APIService object will look like:

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1alpha1.wardle.k8s.io
spec:
  insecureSkipTLSVerify: true
  group: wardle.k8s.io
  groupPriorityMinimum: 1000
  versionPriority: 15
  service:
    name: api
    namespace: wardle
  version: v1alpha1

Leaving aside the more advanced configuration details, this object will instruct the kube-aggregator to forward v1alpha1.wardle.k8s.io requests to the API extension service, implemented by a pod in the wardle namespace.

Now that we have covered the basic concepts, we can deploy a Horizontal Pod Autoscaler using Kubernetes custom metrics.

Kubernetes autoscaler using Sysdig’s custom metrics

Configuring your HPA with custom metrics is pretty straightforward thanks to KEDA, which automatically extends the Kubernetes API metrics endpoint for you. This allows you to use Prometheus PromQL queries to trigger your HPA with just a simple configuration.

Check out how to trigger a Kubernetes autoscaler with Prometheus metrics, and even easier, with Sysdig metrics!

Conclusion

One of the strong points of Kubernetes has always been its extensibility. Thanks to the aggregation layer, you can extend the API, without adding extra complexity or configuration to the resource consumers (the Horizontal Pod Autoscaler in our example).

If you plan to use this integration in your organization, or just a lab environment, we definitely want to hear from you! You can reach us using slack or twitter and, of course, PRs to the project are welcome.

If you would like to run this example, but don’t have a Sysdig Monitor account, we invite you to sign-up for a free trial.

The post Kubernetes pod autoscaler using custom metrics appeared first on Sysdig.

Kubernetes security tools – Categories

• Kubernetes image scanning and static analysis
• Anchore
• Clair
• Dagda
• KubeXRay
• Snyk
• Trivy
• Kubernetes runtime security
• Falco
• Linux runtime security frameworks
• Sysdig open source
• Kubernetes network security
• Aporeto
• Calico
• Cilium
• Istio
• Tigera
• Trireme
• Image distribution and secrets management
• Grafeas
• In-toto
• Portieris
• Vault
• K8s security audit
• Kube-bench
• Kube-hunter
• Kubeaudit
• Kubesec
• Open Policy Agent
• End-to-end K8s security commercial products
• Aqua Security
• Capsule8
• Cavirin
• Google SCC
• Layered Insight (Qualys)
• Neuvector
• StackRox
• Sysdig Secure
• Tenable Container Security
• Twistlock (Palo Alto)

Kubernetes image scanning

Anchore

Clair

Dagda

• It natively integrates with ClamAV, doubling down not only as a container image scanner but also antivirus software.
• Dagda also provides runtime protection features, gathering real time events from the Docker daemon and integrating with CNCF’s Falco to collect runtime container security events.

KubeXray

Snyk

Trivy

Kubernetes runtime security

Falco

Linux runtime security frameworks

Sysdig open source

Kubernetes network security

Aporeto

Calico

Cilium

Istio

Tigera

Trireme

Image distribution and secrets management

Grafeas

• Who built and signed a specific container?
• Did it pass all the security scanners and policy checks? When? What was the output for this tools?
• Who deployed it in production? What were the specific parameters used to deploy?

In-toto

Portieris

Vault

Security audit

Kube-bench

Kube-hunter

Kubeaudit

Kubesec

Open Policy Agent

End-to-end commercial security tools

Aqua Security

• Image scanning, integrated with your container registry or CI/CD pipeline
• Runtime protection to detect container modifications or suspicious activity
• Container-native application firewall
• Serverless security for cloud services
• Compliance and audit reporting, integrated with event logging

Capsule8

Cavirin

Google Cloud Security Command Center

Layered Insight (Qualys)

NeuVector

StackRox

Sysdig Secure

• ML based runtime profiling and anomaly detection
• Runtime policies based on system events, K8s-audit API, community (FIM, cryptojacking) and MITRE ATT&CK framework
• Incident remediation and response

Tenable Container Security

Twistlock (Palo Alto Networks)

Help us build the best security tools directory!

The post 33(+) Kubernetes Security Tools appeared first on Sysdig.

Kubernetes 1.15 – Editor’s pick:

• #95 CustomResourceDefinitions
• #383 Redesign event API
• #962 Execution hooks
• #624 Scheduling framework

Kubernetes 1.15 core

#1024 NodeLocal DNSCache

#383 Redesign event API

#492 Admission webhook

#624 Scheduling framework

#606 Support 3rd party device monitoring plugins

#757 Pid limiting

#902 Add non-preempting option to PriorityClasses

#917 Add go module support to k8s.io/kubernetes

#956 Add Watch bookmarks support

#962 Execution hooks

• Application-consistency snapshotting
• Upgrade
• Prepare for some lifecycle event like a database migration
• Reload a config file
• Restart a container

apiVersion: apps.k8s.io/v1alpha1
kind: HookAction
metadata:
  name: action-demo
Action:
  exec:
    command: ["run_quiesce.sh"]
  actionTimeoutSeconds: 10

#981 PDB support for custom resources with scale subresource

• Concern: don’t reduce serving capacity by more than 10%.
• Solution: use PDB with minAvailable 90%.

Kubernetes 1.15 custom resources

#95 CustomResourceDefinitions

• Structural schema using OpenAPI
• CRD pruning
• CRD defaulting
• Webhook conversion moved to beta
• Publishing the CRD OpenAPI schema

#692 Publish CRD OpenAPI schema

#575 Defaulting and pruning for custom resources

preserveUnknownFields: false
validation:
  openAPIV3Schema:
    type: object

spec:
  type: object
  properties:
    cronSpec:
      type: string
      pattern: '^(d+|*)(/d+)?(s+(d+|*)(/d+)?){4}

#598 Webhook conversion for custom resources


Stage: Graduating to Beta

Feature group: API

Different CRD versions can have different schemas. You can now handle on-the-fly conversion between versions defining and implementing a conversion webhook. This webhook will be called, for example, in the following cases:



 	
custom resource is requested in a different version than stored version.


 	
Watch is created in one version but the changed object is stored in another version.


 	
custom resource PUT request is in a different version than storage version.


 	





There is an example implementation of a custom resource conversion webhook server that you can use as a reference.

Configuration management


#515 Kubectl get and describe should work well with extensions


Stage: Graduating to Stable

Feature group: Cli

Now it is possible for third party API extensions and CRDs to provide custom output for kubectl get and describe. This moves the output printing to the server side allowing for better extensibility and decoupling the kubectl tool from the specifics of the extension implementation.

You can read the design proposal for this feature and the related server-side get here.

#970 Kubeadm: New v1beta2 config format


Stage: Graduating to Beta

Feature group: Cluster lifecycle

Over time, the number of options to configure the creation of a Kubernetes cluster has greatly increased in the kubeadm config file, while the number of CLI parameters has been kept the same. As a result, the config file is the only way to create a cluster with several specific use cases.

The goal of this feature is to redesign how the config is persisted, improving the current version and providing a better support for high availability clusters using substructures instead of a single flat file with all the options.

#357 Ability to create dynamic HA clusters with kubeadm


Stage: Graduating to beta

Feature group: Cluster lifecycle

Kubernetes can have more than a single control plane to provide high availability. The kubeadm tool is now able to set up a Kubernetes HA cluster:



 	
With stacked control plane nodes, where etcd nodes are colocated with control plane nodes


 	
With external etcd nodes, where etcd runs on separate nodes from the control plane


 	





This feature was introduced as a net new alpha in Kubernetes 1.14, you can read about the motivation, design and use cases covered in this KEP.

Cloud providers


#423 Support AWS network load balancer


Stage: Graduating to Beta

Feature group: AWS

You can now use annotations in your Kubernetes 1.15 services to request the new AWS NLB as your LoadBalancer type:

metadata:
      name: my-service
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
Unlike Classic Elastic Load Balancers, Network Load Balancers (NLBs) forward the client's IP through to the node. The AWS network load balancer has been supported in Kubernetes as Alpha feature since v1.9. The code and API has been stabilized, reaching beta stage.
#980 Finalizer protection for service LoadBalancers
Stage: Alpha

Feature group: Network

By default, the LoadBalancer resources allocated in a cloud provider should be cleaned-up after the related Kubernetes service is deleted.

There are, however, various corner cases where cloud resources are orphaned after the

associated Service is deleted. Finalizer Protection for Service LoadBalancers was

introduced to prevent this from happening.

This finalizer will be attached to any service that has type=LoadBalancer if the cluster has enabled cloud provider integration. Upon the deletion of such service, the actual deletion of the resource will be blocked until this finalizer is removed.
Storage
#625 In-tree storage plugin to CSI Driver Migration
Stage: Alpha

Feature group: Storage

Storage plugins were originally in-tree, inside the Kubernetes codebase, increasing the complexity of the base code and hindering extensibility.

The target is to move all this code to loadable plugins that can interact with Kubernetes through the Container Storage Interface.

This will reduce the development costs and will make it more modular and extensible, increasing compatibility between different versions of the storage plugin and Kubernetes code base. You can read more about the ongoing effort to move storage plugins to CSI here.
#989 Extend allowed PVC DataSources
Stage: Alpha

Feature group: Storage

Using this feature, you can “clone” an existing PV.

Clones are different than Snapshots. A Clone results in a new, duplicate volume being provisioned from an existing volume -- it counts against the users volume quota, it follows the same create flow and validation checks as any other volume provisioning request, it has the same lifecycle and workflow.

You need to be aware of the following when using this feature:

 	
Cloning support VolumePVCDataSource is only available for CSI drivers.
 	
Cloning support is only available for dynamic provisioners.
 	
CSI drivers may or may not have implemented the volume cloning functionality.
 	


#1029 Quotas for ephemeral storage
Stage: Alpha

Feature group: Node

The current quota mechanism relies on periodically walking each ephemeral volume, this method is slow, and has high latency.

The mechanism proposed in this feature utilizes filesystem project quotas to provide monitoring of resource consumption and optionally enforcement of limits.

It’s goals are:

 	
Improve performance of monitoring by using project quotas in a non-enforcing way to collect information about storage utilization of ephemeral volumes.
 	
Detect storage used by pods that is concealed by deleted files being held open.
 	


This could also provide a way to enforce limits on per-volume storage consumption by using enforced project quotas.
#531 Add support for online resizing of PVs
Stage: graduating to Beta

Feature group: Storage

This feature enables users to expand a volume's file system by editing a PVC without having to restart the pod using the PVC.

Expanding in-use PVCs is a beta feature and is enabled by default via ExpandInUsePersistentVolumes feature gate.

File system expansion can be performed:

 	
When pod is starting up
 	
When pod is running and underlying file system supports online expansion (XFS, ext3 or ext4).
 	


Read more about this feature in the Kubernetes 1.15 official documentation.
#559 Provide environment variables expansion in sub path mount
Stage: Graduating to Beta

Feature group: Storage

Systems often need to define the mount paths depending on env vars. The previous workaround was to create a sidecar container with symbolic links.

To avoid boilerplate, they are going to introduce the possibility to add environment variables to the subPath, so instead of writing:
env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
 
   ...
   
    volumeMounts:
    - name: workdir1
      mountPath: /logs
      subPath: $(POD_NAME)
You could write:
volumeMounts:
    - name: workdir1
      mountPath: /logs
      subPathExpr: $(POD_NAME)
And that’s all, for now! if you enjoy keeping up to date with the Kubernetes ecosystem, subscribe to our container newsletter, a monthly email with the coolest stuff happening in the cloud-native ecosystem.

default: "5 0 * * *"

#598 Webhook conversion for custom resources

• custom resource is requested in a different version than stored version.
• Watch is created in one version but the changed object is stored in another version.
• custom resource PUT request is in a different version than storage version.

Configuration management

#515 Kubectl get and describe should work well with extensions

#970 Kubeadm: New v1beta2 config format

#357 Ability to create dynamic HA clusters with kubeadm

• With stacked control plane nodes, where etcd nodes are colocated with control plane nodes
• With external etcd nodes, where etcd runs on separate nodes from the control plane

Cloud providers

#423 Support AWS network load balancer

#980 Finalizer protection for service LoadBalancers

Storage

#625 In-tree storage plugin to CSI Driver Migration

#989 Extend allowed PVC DataSources

• Cloning support VolumePVCDataSource is only available for CSI drivers.
• Cloning support is only available for dynamic provisioners.
• CSI drivers may or may not have implemented the volume cloning functionality.

#1029 Quotas for ephemeral storage

• Improve performance of monitoring by using project quotas in a non-enforcing way to collect information about storage utilization of ephemeral volumes.
• Detect storage used by pods that is concealed by deleted files being held open.

#531 Add support for online resizing of PVs

• When pod is starting up
• When pod is running and underlying file system supports online expansion (XFS, ext3 or ext4).

#559 Provide environment variables expansion in sub path mount

The post What’s new in Kubernetes 1.15? appeared first on Sysdig.

This GKE security stack is composed of two different deployments:

• Kubernetes Falco agents: You need to install Falco in your cluster to collect, directly from the Kubernetes nodes, the runtime security events and detect anomalous behavior.

• Serverless / Cloud playbooks: A set of Google Cloud Functions that will execute security playbooks (like killing or isolating a suspicious pod) when triggered.

These two pieces will communicate using Google Pub/Sub.

Before describing the complete GKE security stack, let’s learn more about the building blocks that we intend to use.

Introducing Falco for GKE security.

Falco is an open source project for container security for Cloud Native platforms such as Kubernetes. Originally developed at Sysdig, it is now an independent project under the CNCF umbrella.

Leveraging Sysdig’s open source Linux kernel instrumentation, Falco gains deep insight into system behavior. The rules engine can then detect abnormal activity and runtime security threats in applications, containers, the underlying host, and the container platform itself.

The Falco engine dynamically loads a set of default and user-defined security rules described using YAML files. This would be a token example of a Falco runtime security rule targeting containers:

- rule: Terminal shell in container
  desc: A shell was used as the entrypoint/exec point into a container with an attached terminal.
  condition: >
    spawned_process and container
    and shell_procs and proc.tty != 0
    and container_entrypoint
  output: >
    A shell was spawned in a container with an attached terminal (user=%user.name %container.info
    shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline terminal=%proc.tty)
  priority: NOTICE
  tags: [container, shell]

The rule above will detect and notify any attempt to attach a terminal shell in to a running container. This shouldn’t happen (or at least not in the production, user-facing infrastructure). It is out of scope for this article to fully explain the falco rule format or capabilities, so we are just going to use the default security ruleset. We just wanted to let you know that you can create and customize your own container security rules if you want to.

Also note that Falco is not just using the kernel instrumentation datasource, but can also consume security-related events from other runtime sources, like the Kubernetes Audit Log. This will be an example of a Falco rule designed to detect unwanted Kubernetes ClusterRole tampering:

- rule: System ClusterRole Modified/Deleted
  desc: Detect any attempt to modify/delete a ClusterRole/Role starting with system
  condition: kevt and (role or clusterrole) and (kmodify or kdelete) and (ka.target.name startswith "system:") and ka.target.name!="system:coredns"
  output: System ClusterRole/Role modified or deleted (user=%ka.user.name role=%ka.target.name ns=%ka.target.namespace action=%ka.verb)
  priority: WARNING
  source: k8s_audit
  tags: [k8s]

GKE security stack: Falco, Pub/Sub and Cloud Functions.

In order to extend Falco, we are going to integrate two Google Cloud technologies as part of our stack: Google Cloud Functions and Google Cloud Pub/Sub. 

Google Cloud Functions is a lightweight compute solution for developers to create single-purpose, stand-alone functions that respond to Cloud events without the need to manage a server or runtime environment. We will use this serverless approach to implement security playbooks, in other words, automated remediation actions based on the event data and container metadata coming from the Falco engine.

Asynchronous communication between a set of producers and consumers requires an efficient and reliable messaging middleware. These solutions are commonly known as Publish/Subscribe messaging (PubSub). By using Google Cloud Pub/Sub we can abstract away all the complexities associated with the communication of our two major building blocks.

We can now put everything together using the following diagram:

Listed by alphabetical order:

A – GKE cluster and Kubernetes nodes

This is the Kubernetes cluster that you want to monitor and secure. This is where the entire GKE security workflow starts (detecting a security threat) and ends (performing a remediation action).

B – Falco deployment using a DaemonSet

Falco will be deployed as a Kubernetes DaemonSet, which means that you will have a Falco pod Running in each Kubernetes node. This pod is composed of two containers “C” and “D”.

C – Falco Cloud Native runtime security engine

The Falco security engine that we described above running inside a container. It will keep monitoring the host and container activity and forwarding the triggered security events to container “D”.

Let’s take a look at this security event message generated by Falco:

{  
   "output":"17:10:29.724747835: Notice A shell was spawned in a container with an attached terminal (user=root nginx1 (id=06b29170462e) shell=bash parent=<NA> cmdline=bash  terminal=34816)",
   "priority":"Notice",
   "rule":"Terminal shell in container",
   "time":"2019-03-28T17:10:29.724747835Z",
   "output_fields":{  
      "container.id":"06b29170462e",
      "container.name":"nginx",
      "evt.time":1553793029724747835,
      "proc.cmdline":"bash ",
      "proc.name":"bash",
      "proc.pname":null,
      "proc.tty":34816,
      "user.name":"root"
   }
}

We have all the relevant pieces of information that we need:

• The security rule that was fired
• Affected container (and its ID)
• Timestamp for the event
• Trespassing user and process id

These events are then passed to the connector sidecar container “D”.

D – Falco Pub/Sub sidecar connector

This container receives and processes the JSON payloads for the security events before sending them to the Google Cloud Pub/Sub queue.

E – Google Cloud Pub/Sub message broker

This middleware piece manages the connection between the Falco event emitters (B) and the serverless function consumers (F).

Thanks to this mediation we can replace any of the two parts separately, event if the other end is not yet ready to send or accept messages and also enables us to plug different event consumers with different purposes in the future, if we want to do so.

F – Security playbooks implemented as Google Cloud Functions

The Google Cloud Functions bring us the opportunity to implement security functions without worrying about supporting infrastructure or maintenance.

They can implement different security playbooks:

• Kill the offending pod
• Create a Sysdig capture that will allow you to perform advanced forensics over the security incident
• Isolate the pod from the network
• Forward a Slack notification
• Taint the node where the security event manifested
• … more to come.

These functions are subscribed to a PubSub topic, their operational workflow can be summarized as:

• The PubSub subscription hook notifies the function(s) whenever there is a new security event waiting in the pipe
• Every individual function parses the event and decides whether to fire the response depending on the event type and container metadata
• If the function is triggered, it will contact the Kubernetes API (H) and perform the configured remediation actions

G – Kubernetes API server

GKE provides a Kubernetes API endpoint that will authorize and accept the Google Cloud function request. This way the actions will be executed in the cluster (ie, the offending pod will be effectively killed), closing the cycle.

H – Google Cloud Security Command Center

The Falco security engine can also forward the security events and alerts once they are translated into Google Cloud Security Command Center findings. With just this simple integration, GCSCC will double down as the SIEM for your Falco runtime engine.

How to implement GKE security with Falco.

Enough theory! Let’s install the GKE security stack already.

You need to have:

• A Kubernetes cluster, created using the Google Kubernetes Engine
• The gcloud cli tool, initialized and configured to connect to your Google Cloud account
• The kubectl cli tool configured to interact with your GKE Kubernetes cluster
• A terraform available if you are using the automation we have developed to quickly create the Google Pub/Sub topic
• A clone of the Kubernetes Response Engine repository

Creating needed infrastructure on GKE

In Sysdig we are people obsessed with the automation. In this case, creating a Pub/Sub topic is not a big deal, but we crafted a Terraform manifest to automate this task.

First of all, clone the repository:

$ git clone https://github.com/falcosecurity/kubernetes-response-engine.git

Next step is to access the deployment directory and choose the google-clouddirectory and just type makefor deploying the Pub/Sub topic.

$ cd kubernetes-response-engine/deployment/google-cloud
$ make

Of course, you will need to specify the Google Cloud project required settings. Our recommendation is to use the environment variables and with something like direnv you can keep the values for further references. If you feel more comfortable using the Google Cloud Console tool, it also will work fine.

Installing Falco on GKE

As we mentioned before, this software stack is composed of two main building blocks. We will start installing the software living inside the cluster, the Falco DaemonSet.

We have several options to deploy Falco on GKE but this time, we want to do it using the Google Marketplace. Once we access to the Marketplace and select the Falco Kubernetes application, we can configure Falco to be deployed in our GKE cluster:

We need to put attention on the following fields:

• Enable Pub/Sub output for Falco: It must be enabled in order to allow Falco to send alerts to Pub/Sub.
• Pub/Sub topic where Falco is going to send alerts: name for the PubSub topic that will connect Falco with the Google Cloud Functions, you can use any name, for example falco-pubsub, remember the name you use here, because you will need to connect the functions to the same topic.
• Project where Pub/Sub topic is deployed: The Google Cloud project that is hosting your Kubernetes cluster, you can check the project ID from your Google Cloud web console.
• Service Account credentials used to publish aerts from Falco to a Pub/Sub topic: In order to authenticate with several google cloud services, we need to provide an identity. In this case we need authorization to send messages to a Pub/Sub topic. You can get more information about how to create and manage service accounts keys in Google Cloud Documentation. Once you have the JSON file, you can copy it in the field or if you prefer you can encode it with base64 and copy its content in the field.

A few minutes after deploying Falco, you will be able to see their pods running (one per node):

$ kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
sysdig-falco-1-mwv2j   2/2     Running   1          2m
sysdig-falco-1-t2prc   2/2     Running   1          2m
sysdig-falco-1-zldz2   2/2     Running   1          2m

As you can see, there are two containers per pod (READY column). The in-cluster part is ready, now let’s deploy the serverless entities.

Deploying security playbooks for GKE as Google Cloud Functions

As long as we already cloned the kubernetes-response-engine repository, we need to change the directory to the playbooks one:

$ cd kubernetes-response-engine/playbooks

You can list the different serverless functions available in this repository:

$ ls functions/ 
capture.py  delete.py  demisto.py  isolate.py  phantom.py  slack.py  taint.py

So we are going to deploy one of these functions to check the operation of our GKE security stack. You can see a full list of the available functions with their purpose and input parameters here.

The function we are going to deploy is the deleteone. We provide a shell script that will take care of this, you only have to type one command:

$ ./deploy_playbook_gke -p delete -t falco-alerts -s falco.notice.terminal_shell_in_container -k <cluster_name> -z <gcloud_zone> -n <gcloud_project>

• cluster_name: The Kubernetes cluster name which we are going to securize.
• gcloud_zone: The Google Cloud default zone to manage resources in. You can use the same zone that the one where your Kubernetes cluster is deployed,
• gcloud_project:The Google Cloud project that is hosting your Kubernetes cluster, you can check the project ID from your Google Cloud web console.

You can find all these fields in your Google Cloud Console:

And when you run the script you should see something like:

Fetching cluster endpoint and auth data.
kubeconfig entry generated for sysdig-work.
Deploying function (may take a while - up to 2 minutes)...done.
availableMemoryMb: 256
entryPoint: handler
environmentVariables:
  KUBECONFIG: kubeconfig
  KUBERNETES_LOAD_KUBE_CONFIG: '1'
  SUBSCRIBED_ALERTS: falco.notice.terminal_shell_in_container
eventTrigger:
  eventType: google.pubsub.topic.publish
  failurePolicy: {}
  resource: projects/<gcloud_project>/topics/falco-alerts
  service: pubsub.googleapis.com
labels:
  deployment-tool: cli-gcloud
name: projects/<gcloud_project>/locations/us-central1/functions/delete
runtime: python37
serviceAccountEmail: <gcloud_project>@appspot.gserviceaccount.com
sourceUploadUrl: [...]                                                                                                                                        
status: ACTIVE
timeout: 60s
updateTime: '2019-04-10T15:17:24Z'
versionId: '1'

And that’s all, you should be able to see it in Google Console:

GKE security playbooks in action!

To test these functions, you can create a random victim pod:

$ kubectl run --generator=run-pod/v1 nginx-falco --image=nginx        
pod/nginx-falco created

$ kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
nginx-falco            1/1     Running   0          5s

Now, we simulate one of the security incidents spawning an interactive shell in this container:

$ kubectl exec -it nginx-falco bash                           
root@nginx-falco:/# command terminated with exit code 137

As you can see, the bash process and the pod itself were terminated.

Let’s check the logs produced by the Falco engine:

$ kubectl logs -l role=security -c falco | grep "nginx-falco"
{"output":"16:16:47.908565738: Notice A shell was spawned in a container with an attached terminal (user=root k8s.pod=nginx-falco container=0042056722bb shell=bash parent=<NA> cmdline=bash  terminal=34816)","priority":"Notice","rule":"Terminal shell in container","time":"2019-04-01T16:16:47.908565738Z", "output_fields": {"container.id":"0042056722bb","evt.time":1554135407908565738,"k8s.pod.name":"nginx-falco","proc.cmdline":"bash ","proc.name":"bash","proc.pname":null,"proc.tty":34816,"user.name":"root"}}

Now, click on your function name (delete) and check the Logs associated to the function execution:

Our GKE security pipeline is working and we have full logs for the entire operation!

GKE Hackers, welcome to Falco :)

To avoid making this post too extensive we have just used the default set of Falco rules, and the simple “terminal shell in container” example. But there is so much more you can do with the Falco engine:

• Targeting specific CVEs shortly after they have been published, immediately protecting your infrastructure.
• Parsing the Kubernetes audit log to detect abnormal operations at the cluster level, performed by the Kubernetes users or a service account related to a software entity.
• Write your own security playbooks extending the playbooks library and functions available in the Falco repository and deploy them as serverless functions.

If you have made any modifications to this GKE security stack or are just using it and would like to share some feedback, we would love to hear about your experience!

The post GKE security using Falco, Pub/Sub, and Cloud Functions appeared first on Sysdig.

How StatsD protocol works

• Importing the StatsD client library: import statsd
• Declaring the StatsD variables that we want to report, specifying their metric type: counter = client.get_counter(“counter”)
• Periodically emitting a metric value update: average.send(‘avg’, 5)

• You need to inject the minimum network configuration required to find the StatsD server. This means dealing with volatile IPs, network segmentation and routing, firewalls, etc.
• The information about which entity (host, pod, container, process) originally sent the metric is lost unless you format the metrics to explicitly store this data.

This is especially problematic if you want to deploy containers and microservices, where the software entities are volatile by nature and you need to group them by functionality (for example Kubernetes deployments and services) to make sense of the data.

• The StatsD server becomes a single point of failure. If it crashes, all the monitoring process is temporarily down and the UDP packages for that period are lost.

Types of StatsD metrics

Gauge

<metric name>:<value>|g

<metric name>:+<value>|g

<metric name>:-<value>|g

Counters

<metric name>:<value>|c[|@<rate>]

Timers

<metric name>:<value>|ms[|@<rate]

Histograms

<metric name>:<value>|h

Meters

<metric name>:<value>|m

<metric name>

Sets

<metric name>:<value>|s

unique_users:203|s

unique_users:340|s

unique_users:203|s

StatsD code examples

Golang code instrumentation with StatsD

client := statsd.NewStatsdClient(url, prefix)

client.CreateSocket()

File: statsd-go.go
------------------

package main

import (
  "github.com/quipo/statsd"
  "math/rand"
  "time"
)

func main() {
  rand.Seed(time.Now().Unix())

  // Create the client
  client := statsd.NewStatsdClient("127.0.0.1:8125", "golang.")
  defer client.Close()
  // Connect to the statsd server
  err := client.CreateSocket()
  if err != nil {
     panic(err)
  }

  for {

     // Gauge
     client.Gauge("foo", rand.Int63n(100))

     // Counter
     client.Incr("bar", 1)

     // Counter with sampling
     client.IncrWithSampling("bar_sampled", 5, 0.1)

     // Timer
     client.Timing("timed", rand.Int63n(100000))

     // Wait for a random time between 500 and 1000 ms before sending the data again.
     time.Sleep(time.Duration(int(time.Millisecond) * (rand.Intn(500) + 500)))
  }

}

client := statsd.NewStatsdClient("127.0.0.1:8125", "golang.")

$ netcat -ul 8125
golang.foo:46|ggolang.bar:1|cgolang.timed:58840|msgolang.foo:6|ggolang.bar:1|cgolang.timed:78230|msgolang.foo:2|ggolang.bar:1|cgolang.timed:9045|msgolang.foo:30|ggolang.bar:1|cgolang.timed:57087|msgolang.foo:45|ggolang.bar:1|cgolang.timed:15685|ms

Node.js / Javascript code instrumentation with StatsD

$ npm install --save statsd-client

File: statsd-js.js
------------------

const SDC = require('statsd-client');
const sdc = new SDC({host: '127.0.0.1'});

setInterval(() => {
   const timer = new Date();

   // Increment counter by one.
   sdc.increment('node.counter');
  
   // Increment counter by 10
   sdc.increment('node.counter_10', 10);

   // Set gauge to 10
   sdc.gauge('node.gauge', 10);

   // Calculates time diff of time between the variable and
   // when the function was called
   sdc.timing('node.timer', timer);

   // Set will count just 2 elements since '50' is repeated
   sdc.set('node.set', 50);
   sdc.set('node.set', 100);
   sdc.set('node.set', 50);

   // Histogram with tags
   sdc.histogram('node.histogram', 10, {foo: 'bar'});
}, 1000);

$ netcat -ul 8125
node.counter:1|c
node.counter_10:10|c
node.gauge:10|g
node.timer:1|ms
node.set:50|s
node.set:100|s
node.set:50|s
node.histogram:10|h|#foo:barnode.counter:1|c

Python code instrumentation with StatsD

 $ pip install python-statsd

File: statsd-python.py
----------------------

import statsd
import time

if __name__ == '__main__':     
   # Create a new connection for the client
   connection = statsd.Connection(
       host='127.0.0.1',
       port=8125,
       sample_rate=1,
   )
  
   # Create the client
   client = statsd.Client("python", connection)
  
   # Create counter
   counter = client.get_counter("counter")
  
   # Create gauge
   gauge = client.get_gauge("gauge")
  
   # Create average
   average = client.get_average("average")
  
  
   while True:
       # Create a timer
       timer = client.get_timer("timer")
  
       # Will send the elapsed time once all the block has been executed
       with timer:
           counter += 1  # Increment by one the counter
  
           gauge.set('foo', 10) # Send a gauge of 10
  
           average.send('avg', 5)
  
       time.sleep(1)

$ netcat -ul 8125
python.counter:1|cpython.gauge.foo:10|gpython.average.avg:5|apython.timer.total:0.07200241|mspython.counter:1|cpython.gauge.foo:10|gpython.average.avg:5|apython.timer.total:0.29730797|mspython.counter:1|cpython.gauge.foo:10|gpython.average.avg:5|apython.timer.total:0.30040741|mspython.counter:1|cpython.gauge.foo:10|gpython.average.avg:5|apython.timer.total:0.29134750|ms

StatsD code in Java – Deprecated client

StatsD instrumentation and Docker containers with Sysdig Monitor

• Origin of the metrics (container, pod, deployment, service) is lost if you don’t explicitly append it to the metric
• Entities are ephemeral appearing, disappearing being scaled up and down, manual tagging and labelling is just not feasible anymore
• Network configuration and visibility requires injecting extra parameters in your containers at boot time
• Adding a separate process, inside the same container or as a sidecar container, to aggregate and push the StatsD metrics is doable but goes against the inherent simplicity of containers

• Getting rid of all the network configuration, routing and maintenance
• Super simple “push to localhost” approach
• Out of the box host / container / orchestrator (Docker, Kubernetes, OpenShift) tagging
• Provides a monitoring backend including long term storage, dynamic aggregation based on labels and flexible dashboarding and visualization

Conclusions

The post Monitoring StatsD: metric types, format + code examples appeared first on Sysdig.