We take pride that our customers report massive improvements in mean time to resolution when they’re troubleshooting (think 10X+). Along these lines, to help teams respond quickly, we’ve integrated with PagerDuty for alert notifications.

Our integration with PagerDuty is one of the most used integrations within the Sysdig platform. By hooking up your Sysdig account to your PagerDuty account, you can automatically route Sysdig-generated notifications directly through PagerDuty. This lets you manage your on-call duty and route all your alerts accordingly, from one central location.

We’ve taken our integration with PagerDuty one step further by linking notification status across the two apps. Now, Sysdig doesn’t just push notifications to PagerDuty, we’ll also automatically keep the status of that notification up to date!

Status of notifications in Sysdig Monitor is tracked in two ways:

• First off, if a threshold condition resolves to an acceptable state, then a notification will automatically be moved to an “OK” status. This way you still have a record of the issue, but you know that your system is no longer in an alert-worthy state.
• Second, you can manually resolve any notification as well. This way, if you’ve investigated a situation, or are otherwise not concerned for any reason, you can dismiss the notification, and clear up your alert dashboard.

You can of course sort all of your notifications by status within Sysdig Monitor, so that only the most important will filter to the top:

And now, with this feature, if any notification is resolved in Sysdig Monitor, we will automatically push the resolved status of the alert to PagerDuty. This way you can keep your PagerDuty notifications list clean, and not have to interface with the same notification in multiple places.

The design of this feature is based on conversations we’ve had with many current Sysdig Monitor users about their preferred workflows with PagerDuty. We’re always looking to improve though, and will be continuing to add new features based on feedback going forward.

If you’d like to give Sysdig a shot, please don’t hesitate to sign up for a free trial above! If you’re already a PagerDuty user, you’ll hit the ground running.

The post Sysdig and PagerDuty: a Superior Alerting Experience appeared first on Sysdig.

In this blog we will dive deeper into how to integrate Sysdig Secure with ACR to scan images for Kubernetes for security and compliance purposes. Sysdig has offered unified visibility and security for container and Kubernetes deployments on Azure for years. Now with native CI/CD and registry integrations, you can shift security visibility earlier into your build pipeline.

About Azure Container Registry

ACR provides a container image registry that allows you to store container images for all types of orchestration platforms including Kubernetes, Docker, and Red Hat OpenShift as well as Azure services such as App Service, Machine Learning, and Batch. ACR handles private Docker container images as well as related content formats, such as Helm charts, OCI artifacts, and images built to the OCI image format specification.

Azure Container Registry Security and Sysdig Secure

Scanning images in Azure Container Registry is the same as scanning from any other Docker v2 compatible registry. Once configured, the entire registry or individual images and tags can be analyzed and then evaluated against a Sysdig Secure Scanning policy.

The first step is to pass the ACR credentials into Sysdig Secure to give access to the registry. Once configured the Sysdig Secure scanning engine can pull any image stored within the registry into the engine for analysis.

When an image is pulled into the scanning engine Sysdig Secure will provide visibility into:

• Official OS packages
• Unofficial packages
• Configuration files
• Artifacts such as Javascript NPM modules, Python PiP, Ruby GEM, and Java JAR archives
• Secrets, credentials like tokens, certificates and other sensitive data
• Known vulnerabilities & available updates

These artifacts are then stored and evaluated against custom scanning policies to spot vulnerabilities, misconfiguration, or compliance issues within your images.

Scanning Container Images in Azure Container Registry

Adding an image to the scanning engine from ACR is as simple as copying the registry URL/image/tag into the Sysdig Secure UI and clicking scan image. This process can also be easily scripted to import all images and to watch repositories for updates.

Once an image has been analyzed a report will be generated that has the outcome of the policy evaluation, all vulnerabilities discovered in OS packages, configuration files, and many other artifacts that are stored for audit and compliance reasons.

This report has details showing knoxsds.azurecr.io/cassandra:latest has passed the policy evaluation, including image metadata, vulnerabilities, and even JAR archives information included in the image.

Conclusion

Hopefully you can see how easy it is to get up and running with both Azure Container Registry and Sysdig Secure. If you’d like to see the integration in action sign up for a trial and follow our blog for more posts about securing containers on Azure.

The post Scanning images in Azure Container Registry appeared first on Sysdig.

Kubernetes audit events

Adds new detections based on audit data from the Kubernetes API

File: Create-Modify-Configmap-With-Private-Credentials.yaml
-----------------------------------------------------------

- macro: contains_private_credentials
  condition: >
    (ka.req.configmap.obj contains "aws_access_key_id" or
     ka.req.configmap.obj contains "aws-access-key-id" or
     ka.req.configmap.obj contains "aws_s3_access_key_id" or
     ka.req.configmap.obj contains "aws-s3-access-key-id" or
     ka.req.configmap.obj contains "password" or
     ka.req.configmap.obj contains "passphrase")
- rule: Create/Modify Configmap With Private Credentials
  desc: >
     Detect creating/modifying a configmap containing a private credential (aws key, password, etc.)
  condition: kevt and configmap and kmodify and contains_private_credentials
  output: K8s configmap with private credential (user=%ka.user.name verb=%ka.verb configmap=%ka.req.configmap.name config=%ka.req.configmap.obj)
  priority: WARNING
  source: k8s_audit
  tags: [k8s]

Sysdig teams

Service based access control

Kubernetes vulnerability management

Admissions controller

Service oriented compliance

Leveraging Kubernetes labels to improve operations and reporting

Security analytics

Integrating metrics for advanced reporting

Wrapping up

The post Introducing Sysdig Secure 2.2: Kubernetes auditing, compliance, and access control. appeared first on Sysdig.

• The classic “dev vs prod” split: Many organizations prefer to limit the number of people accessing data related to their production services. This is about isolating physical infrastructure and the applications on top.
• Microservices where each team needs to only see their own dashboards and field its own alerts: By scoping down the data accessible to individual development teams and their related services, developers can more effectively focus on the information that is relevant to them. This is about building teams based on logical isolation using orchestration or config management metadata.
• Platform as a service where ops teams need to see the entire platform: This is somewhat the flip of the previous use case, enabling certain people to see all data for all services as well as the underlying hardware. This is perfect for managed service providers who are managing a multi-tenant environments, or devops teams using a similar model within their own organization.
• Restricted environments: An even more specific use case of the microservices example to limit data access for security and compliance: Certain services, such as authentication and billing, may have a very specific set of individuals authorized to access them.
• Organizations or environments that need to segment incident response for efficiency: This is a wide ranging use case. We’ve seen very large organizations form teams just to simplify access; smaller organizations create ephemeral teams to troubleshoot a particular issue; or teams formed to optimize QA & support access to system data.

How to set up Sysdig Teams

1. Set the scope for a team based on orchestration metadata from Kubernetes, Mesos, and Swarm, as well as other characteristics of your environment.
2. Assign users to any number of teams based on the services or infrastructure they need access to.

Teams in action

Vulnerability management & image scanning

Policy creation

• Defining the images they expect to run within the namespace
• Defining the registry location that all images within that namespace should come from
• Defining which kubernetes deployments within their team should have inbound/outbound connections
• Defining which processes they’d expect to run within the images they have built that are deployed to that namespaces.

The post Service based access control with Sysdig Secure Teams. appeared first on Sysdig.

Vulnerability Management

• Official OS Packages
• Unofficial OS Packages
• Configuration Files
• Language Modules – NPM, PiP, GEM, and Java Archives
• Image metadata & more

Compliance

Security Analytics

What’s next?

The post Sysdig Secure 2.0 – adds vulnerability management, 200+ compliance checks, and security analytics. appeared first on Sysdig.

• The Falco wiki – check out the default rules and learn the basics of lists, macros, etc
• The Sysdig wiki – learn the filtering syntax that falco rules use to detect unwanted behavior
• Blog – Getting started writing falco rules
• Blog – Friends don’t let friends curl | bash
• A Katacoda scenario that goes through the steps of creating and triggering falco rules.

Name the Web Fetch Programs

list: web_fetch_binaries
items: [curl, wget]

macro: web_fetch_programs
condition: (proc.name in (webfetchbinaries))

Capture Spawning a Web Fetch Program

macro: spawned_process
condition: evt.type = execve and evt.dir=<

macro: spawn_web_fetcher
condition: (spawned_process and web_fetch_programs)

Add exclusions, define the rule

macro: allowed_web_fetch_containers
condition: (container.image startswith quay.io/my-company/services)

Create Policy Associated With Rule

Verify Policy Is Working

The post Auditing container activity – A real example with wget and curl using Sysdig Secure. appeared first on Sysdig.

Updated Look & Feel of the Sysdig Intelligence Platform

Kubernetes Security for the App Developer

Default Policies mapped to Compliance Frameworks

Increased Investigation efficiency

Sysdig Secure + Cloud Security Command Center

Conclusion

• Container and Kubernetes compliance.
• Visit Sysdig Brightalk Channel for sessions on best practices securing containers.

The post Sysdig Secure 1.5 – enhanced kubernetes security & compliance; integrates with Google Cloud. appeared first on Sysdig.

Kubernetes Oriented Security

New Kubernetes & AWS Policies

- rule: Contact K8S API Server From Container
  desc: Detect attempts to contact the K8S API Server from a container
  condition: outbound and k8sapiserver and container and not k8s_containers
  output: Unexpected connection to K8s API Server from container (command=%proc.cmdline %container.info image=%container.image connection=%fd.name)
  priority: NOTICE
  tags: [network, k8s, container]

- rule: Contact EC2 Instance Metadata Service From Container
  desc: Detect attempts to contact the EC2 Instance Metadata Service from a container
  condition: outbound and fd.sip="169.254.169.254" and container and not ec2metadatacontainers
  output: Outbound connection to EC2 instance metadata service (command=%proc.cmdline connection=%fd.name %container.info image=%container.image)
  priority: NOTICE
  tags: [network, aws, container]

New Policy Editor

Enterprise Integrations – SSO authentication & new notification channels

• Google Authentication
• SAML
• OpenID (Cloud)
• LDAP (On-premise)

The post Sysdig Secure February Release appeared first on Sysdig.

Let’s recap 2017 by focusing on the viral blogs that climbed up hackernews and into your twitter feeds. We’re going by the data and looking at the blogs that you, the readers visited most in 2017.

1. Container isolation gone wrong

At it’s peak this blog had 164 users coming to the blog per minute. Containers can have separate quotas for CPU, memory, storage… what could possibly go wrong?

One of the main advantages of embracing containers is “lightweight virtualization”. Since each container is just a thin layer around the containerized processes, The user gains enormous efficiencies, for example by increasing the container density per host, or by spinning containers up and down at a very fast pace.

However, as the troubleshooting story in the article will show, this lightweight virtualization comes at the cost of sharing the underlying kernel among all containers, and in some circumstances this can lead to surprising and undesirable effects that container users typically don’t think about.

This troubleshooting tale is rather involved. I’ve started from the basics and worked up to the more complex material in the hope that readers at all levels can get value out of it. Read More

2. Kubernetes monitoring guide

If 2016 was the year of containers, then 2017 was the year of Kubernetes. This blog kicks off a 4 part series covering Kubernetes monitoring, alerting, troubleshooting, and how customers are actually using kubernetes in production.

With over 21,000 stars on GitHub, over a thousand committers, and an ecosystem that includes Google, RedHat, Intel, and more, Kubernetes has taken the container ecosystem by storm. It’s with good reason too: Kubernetes acts as the brain for your distributed container deployment. It’s designed to manage service-oriented applications using containers distributed across clusters of hosts. Kubernetes provides mechanisms for application deployment, service discovery, scheduling, updating, maintenance, and scaling. But what about monitoring Kubernetes? Read More

3. Docker usage report

As someone who’s attended DockerCon for the last 3 years it’s been pretty impressive to see the community grow around containers. This data driven post looks at how people are using Docker in their application environments right now.

Many Docker usage reports are based on user surveys. These are incredibly useful to understand intent and objectives, but can sometimes be inconsistent with how people are actually adopting Docker within their environments right now. The main question we wanted to answer was, “How are people using Docker in their application environments right now?” As the premier container monitoring solution, Sysdig is in a fantastic position to answer this question with actual Docker usage data across hundreds of customers. Read More The top 5 #sysdig viral container blogs of 2017 Click to tweet

4. Sysdig Inspect

Ok, Ok, I know earlier I said we’d keep release posts out of the list, but Sysdig Inspect was too good. It’s not everyday that someone builds an Opensource GUI for system call analysis.

Sysdig is routinely praised by users for the richness of the data it’s able to capture and for the ability to store system, application and container that data into capture files that can be easily shared with other people.

However, extracting insights from rich data can be hard. Mastering the art of analyzing sysdig capture files requires dedication and skills. This is why we constantly try to improve workflows around sysdig and find ways to get better insights with less effort. Today, we bring these efforts to a new level with the release of Sysdig Inspect.

Sysdig Inspect is a powerful, intuitive tool for sysdig capture analysis that runs natively on your Mac or your Linux PC, with a user interface that has been designed for performance and security investigation. Read More

5a. Top 20 Docker security tools

As more and more companies move into production environments with containers, security becomes ever more important. This post rounds up the container security ecosystem through every layer of your security stack, from scanning down to forensics. We are keeping the number “20” in the title, but the list has 22 items at this moment… and growing.

There are quite a few Docker security tools in the ecosystem, how do they compare? This is a comprehensive list of Docker security tools that can help you implement some of the container security best practices.

Is Docker insecure? Not at all. Actually features like process isolation with user namespaces, resource encapsulation with cgroups, immutable images and shipping the minimal software and dependencies reduce the attack vector providing a great deal of protection. But, is there anything else we can do? There is much more than image vulnerability scanning and these are 20 container and Docker specific security tools that can help. Read More

5b. Docker vulnerabilities and threats

A list of container vulnerabilities is great, but it’s much more useful if it includes proof of concept examples how to address them in your environment. This post has been one of the more consistently visited pages since it’s release.

There are some specific parts of Docker based architectures which are more prone to attacks. In this article we are going to cover 7 fundamental Docker security vulnerabilities and threats.

Each section has been divided into:

• Threat description: Attack vector and why it affects containers in particular.

• Docker security best practices: What can you do to prevent this kind of security threats.

• Proof of Concept Example(s): A simple but easily reproducible exercise to get some first hand practice.

Docker vulnerabilities and threats to battle

• Docker host and kernel security
• Docker container breakout
• Container image authenticity
• Container resource abuse
• Docker security vulnerabilities present in the static image
• Docker credentials and secrets
• Docker runtime security monitoring

Bonus content

We’re stacking the deck in 2018 and have already released two great posts to start of the new year.

Making sense of Meltdown/Spectre

The IT world was put on its heels this past week as two of the most significant hardware exploits were announced for CPUs from Intel, ARM, AMD, IBM, and others. Early estimates state that as many as 3 billion devices have this flaw (don’t worry, your Apple II and C64 is safe). As there often is with a vulnerability of this magnitude, there’s a ton of noise associated with the problem, which makes it difficult for the average IT professional to get to the facts. So let’s step back from the hyperbole, and break down exactly what the problem is, and how it affects the day to day life of the average IT professional.

Fishing for miners – cryptojacking honeypots in Kubernetes

We run Kubernetes honeypots so you don’t have to. What would happen if you put a K8s cluster up on AWS but left the API Server port (8080) generally open to the outside world? Have attackers moved on from EC2 exploits to container-specific and kubernetes-specific exploits? One way to find out!

The post Sysdig’s Top Viral Blogs of 2017 appeared first on Sysdig.

Alibaba Cloud has been around for awhile, but with their new $300 trial credit offer there’s no reason not to spin up some instances to see what it’s about.

Not to be confused with AWS ECS Alibaba refers to their compute service as ECS and like GCP, AWS, and Azure they have many services for Databases, load balancers, etc. There’s a ton of different free options, but naturally the first thing I wanted to check out was their container service.

Setting up Alibaba Container Service

The Alibaba Container Service makes it easy to get a handful of hosts booted up, then to schedule and run containerized applications on them. But before we dive into visibility let’s take a look at the building blocks of the Container Service. The first thing we need to do to get started is boot up a cluster.

When building a cluster just click through the GUI to choose your Region, Number of nodes, security group parameters and you’re all set.

Now that you’ve got a small cluster of nodes, let’s deploy some applications. There are a couple different options baked into the Container Service Console

• Docker Images – Choose any image from the docker public repo and with a single click start that container on your service

• Orchestration – use docker compose within the console via orchestration templates, or roll your own orchestrator like kubernetes or mesos

• Applications – pre-built application stacks, or applications that you’ve uploaded to run in containers on the nodes in your cluster

I chose to boot up one of the gitlab applications as well as the wordpress service into the cluster we have running.

Installing Sysdig Monitor

Installing Sysdig Monitor in your cluster is crazy fast. Just copy the docker run command or use existing orchestrator facilities like daemonset in kubernetes to get our agent installed. The Sysdig Monitor agent runs as a container on each node, and then loads a non block kernel module via DKMS to start collecting all system, application, network, and custom metrics (statsd, jmx, prometheus) from your system.

Monitoring Alibaba Container Service

Monitoring Physical Infrastructure: Hosts & Containers

One of the first things to do when you get Sysdig Monitor installed in any environment is to look at a topology map of the cluster. Here we can see every node in the cluster, the cpu usage of the node, and then all ingress and egress connections to other nodes we’re monitoring or external services.

From here we can drill down even further inside the host and see all the containers, their network connections, and then go another layer deeper by drilling down to the process level or even syscall of what’s running inside the container. This provides huge value into understanding the performance of Alibaba Container Service, most monitoring tools only provide container metrics via the docker stats API so you know “hey, my container is running at 98% CPU” but you don’t know why, or what’s running inside that container.

Monitoring Applications running on Alibaba Container Service

Next let’s move to the application layer. In the previous gif we had looked at a host and noticed that there was a MySQL process running inside the wordpress container. Sysdig Monitor comes with hundreds of out of the box integrations with popular applications and services. We’ll automatically recognize the process or protocol and then start pulling metrics from that application and tag them with the metadata from your cloud provider and orchestrator. Such as this example of nginx errors segmented by the kubernetes namespace that they’re coming from.

The same applies to any custom metric from statsd, jmx or prometheus. Just install the agent as a container on the node and we’ll discover everything whether it’s running inside a container or on the node. Let’s check out the gif below to see what this looks like for our containerized MySQL application.

Exploring our Container Service Environment

Next let’s flip over to the explore page of Sysdig Monitor to slice and dice our infrastructure to dynamically troubleshoot and visualize services. The explore page works like an interactive top or pivot table so you can set up and hierarchy and change it on the fly depending on what questions you want to answer about services, hosts, or containers. Here we can look at hosts first, and then switch our grouping up to get the performance of images and the containers that are sharing that image regardless of what physical infrastructure they’re running on.

Conclusion

A couple of slices of pizza and some free credits to learn about monitoring Alibaba Container Service were more than enough to keep me entertained while futzing with different applications on this service.

On the Sysdig Monitor side we’ve still barely touched the surface of the insights you can get into Alibaba Container Service. Check out some of these posts to learn more about how we do alerting, kubernetes monitoring, or more about our custom metrics integrations.

If you’re already running on Alibaba Container Service or any other cloud platform then kick off a 15 node free trial of Sysdig Monitor and let us know what you think!

The post Monitoring Alibaba Container Service appeared first on Sysdig.