Top 15 Kubectl plugins for security engineers

By Nigel Douglas - JANUARY 17, 2023
Topics: Kubernetes & Container Security

SHARE:

Top 15 Kubectl Security Plugins

content:

What are Kubernetes Plugins? DIY Conclusion


Let’s dig deeper into this list of Kubectl plugins that we strongly feel will be very useful for anyone, especially security engineers.

Kubernetes, by design, is incredibly customizable. Kubernetes supports custom configurations for specific use case scenarios. This eliminates the need to apply patches to underlying features. Plugins are the means to extend Kubernetes features and deliver out-of-the-box offerings.

What are Kubernetes Plugins?

Users can install and write extensions for kubectl, the Kubernetes command line tool.

By observing the core kubectl commands as essential building blocks for interacting with a Kubernetes cluster, a cluster administrator can think of plugins as a means of utilizing these building blocks to create more complex behavior.

Plugins extend kubectl with new sub-commands, allowing for new and custom features not included in the main distribution of kubectl.

Why are plugins useful for security operations?

Kubernetes plugins provide countless security benefits to the platform. Incident responders can develop additional functionality “on the fly” in their language of choice.

Since Kubernetes features often fall short in cases where businesses need to achieve “out-of-scope” functionality, teams will often need to implement their own custom operations.

Potential security considerations for Kubernetes plugins

While custom implementations add functionality that is not necessarily provided out-of-the-box with kubectl, these plugins are not always as secure as we would like them to be. This article aims to address the most common or useful Kubernetes plugins for improving your security posture.

Managing plugins with Krew

Krew is a plugin manager maintained by the Kubernetes Special Interest Group (SIG) CLI community. Krew makes it easy to use kubectl plugins and helps you discover, install, and manage them on your machine. It is similar to tools like apt, dnf, or brew. Today, over 200 kubectl plugins are available on Krew – and that number is only increasing. Some projects are actively used and some get deprecated over time, but are still accessible via Krew.

Command to install kubectl plugins via Krew:

kubectl krew install <PLUGIN_NAME>

Kubectl plugins available via the Krew plugin index are not audited, which can cause a problem in the supply chain. As mentioned earlier, the Krew plugin index houses hundreds of kubectl plugins:
https://krew.sigs.k8s.io/plugins/

When you install and run third-party plugins, you are doing this at your own risk. At the end of the day, kubectl plugins are just arbitrary programs running in your shell.

Finally, we want to share our top 15 kubectl plugins that will improve your security posture in your Kubernetes cluster.

1. Stern plugin

Link to GitHub Repository

Stern is a kubectl plugin that works a lot like ‘tail -f’ in Linux. Unlike kubectl log -f, which has its own limitations around input parameters, Stern allows you to specify both the Pod ID and the Container ID as regular expressions.

Any match will be followed and the output is multiplexed together, prefixed with the Pod and Container ID, and color-coded for human consumption (colors are stripped if piping to a file).

You can install Stern with the below Krew command:

kubectl krew install stern

Command to tail an appname in Stern:

kubectl stern appname

This will match any pod containing the word service and listen to all containers within it. If you only want to see traffic to the server container, you could do:

kubectl stern --container

This will stream the logs of all the server containers, even if running in multiple pods.

One interesting security use case for the Stern plugin is to look at authentication activity to your Kubernetes clusters. To show the authentication activity within the last 15 minutes with relevant highlighted timestamps, run the following command:

kubectl stern -t --since 15m auth

2. RBAC-tool

Link to GitHub Repository

Role-based access control (RBAC) is a method of regulating access to computer or network resources based on the roles of individual users within your organization. The RBAC-tool simplifies querying and the creation of RBAC policies.

You can install the RBAC-tool with the below Krew command:

kubectl krew install rbac-tool

If you are unfamiliar with how RBAC roles are assigned to different Kubernetes components, the visualization command generates an insightful graph of all RBAC decisions.

kubectl rbac-tool viz --cluster-context nigel-douglas-cluster

The above command scans the cluster with the kubeconfig context ‘nigel-douglas-cluster.’ These graphs are useful for showing a visual before-and-after of permissions assigned to service accounts.

There are multiple commands other than ‘viz’ provided by the RBAC-tool plugin. The most useful is the ‘who-can’ command. This shows which subjects have RBAC permissions to perform an action denoted by ‘VERB’ (Create, Read, Update, or Delete) on an object.

To see who can read a secret resource by the name ‘important-secret,’ run the below command:

kubectl rbac-tool who-can get secret/important-secret

3. Cilium Plugin

Link to GitHub Repository

Cilium is a network security project that continues to grow in popularity due to its powerful eBPF dataplane. Since Kubernetes is not designed with any specific CNI (Network) Plugin in mind, it can be deciduous trying to manage the Cilium agent via kubectl. That’s why the Cilium team released the Cilium kubectl plugin.

You can install the Cilium plugin with the below Krew command:

kubectl krew install cilium

You can do a connectivity check for a single node powered by Cilium networking via the command:

kubectl cilium connectivity test --single-node <node>

This doesn’t just provide improved operational visibility – it’s incredibly beneficial to network security engineers. For instance, if Cilium is unable to communicate with core components such as ‘Hubble,’ this will show-up in the connectivity test.

Hubble provides network, service, and security observability for Kubernetes. Being able to quickly diagnose a connection error, such as “connection refused,” improves the overall visibility of threats and provides the centralized network event view required to maintain regulatory compliance. If you want to dig deeper into network policies, discover how to prevent a Denial of Service (DoS) attack on Kubernetes.

4. Kube Policy Advisor

Link to GitHub Repository

The kube-policy-advisor plugin suggests PodSecurityPolicies and Open Policy Agent (OPA) Policies for your Kubernetes cluster. While PodSecurityPolicies are deprecated, and therefore should not be used, OPA is very much a recommended tool for admission controller.

You can install advise-policy with the below Krew command:

kubectl krew install advise-policy

This kubectl plugin provides security and compliance checks for Kubernetes clusters. It can help identify potential security risks and violations of best practices in a cluster’s configuration, and provide recommendations for how to remediate those issues. Some examples of the types of checks that kube-policy-advisor can perform include:

  • Ensures pods are running with minimal privileges and are not granted unnecessary permissions.
  • Checks that secrets and other sensitive data are not stored in plain text or checked into source control.
  • Verifies that network policies are in place to protect against unauthorized access to resources.
  • Evaluates the security of container images and ensures that they come from trusted sources.

In Kubernetes, Admission Controllers enforce semantic validation of objects during create, update, and delete operations. With OPA, you can enforce custom policies on Kubernetes objects without recompiling or reconfiguring the Kubernetes API server.

kube-policy-advisor is a tool that makes it easier to create OPA Policy from either a live K8s environment or from a single .yaml file containing a pod specification (Deployment, DaemonSet, Pod, etc.). In the below command, the plugin inspects any given namespace to print a report or OPA Policy.

kubectl advise-policy inspect --namespace=<ns>

Note: If you do not enter a given namespace, it will generate the OPA Policy for all network namespace.

By using kube-policy-advisor, you can help ensure that your Kubernetes cluster is secure and compliant with best practices, which can help protect your applications and data from potential threats.

5. Kubectl-ssm-secret

Link to GitHub Repository

The kubectl-ssm-secret plugin allows admins to import or export their Kubernetes Secrets to or from an AWS SSM Parameter Store path. A Kubernetes Secret is sensitive information – such as a password or access key – that is used within a Kubernetes environment. It’s important to be able to safely control these sensitive credentials when transmitting between Kubernetes and AWS cloud.

You can install the ssm-secret plugin with the below Krew command:

kubectl krew install ssm-secret

Secrets are not unique to Kubernetes, of course. You use Secrets’ data in virtually every type of modern application environment or platform. In the case of the ssm-secret plugin, all parameters found under a given parameter store path can be imported into a single kubernetes secret as “StringData.”

This is incredibly useful if you are reprovisioning clusters or namespaces and need to provision the same secrets over and over. Also, it could be useful to backup/restore your LetsEncrypt or other certificates.

If an AWS parameter at path /foo/bar contains a secret value, and the parameter /foo/passwd contains a secure password, we can view the keys and values in parameter store using the kubectl ssm-secret list subcommand:

kubectl ssm-secret list --ssm-path /foo

Those output parameters can then be imported with the following import command:

kubectl ssm-secret import foo --ssm-path /foo

Security considerations

You must specify a single parameter store path for this plugin to work. It will not recursively search more than one level under a given path. As a result, the plugin is highly opinionated, and users run the risk of failing to import/export secrets to the correct path if they don’t track these paths correctly.

6. Kubelogin

Link to GitHub Repository

If you’re running Kubectl versions v.1.12 or higher, Kubelogin (also known as kubectl-login) is a useful security plugin for logging into clusters via the CLI. It achieves this through OpenID Connect providers like DEX. OpenID Connect is a simple identity layer on top of the OAuth 2.0 protocol. It allows Clients to verify the identity of the End-User based on the authentication performed by an Authorization Server, as well as to obtain basic profile information about the End-User in an interoperable and REST-like manner.

You can install the kubectl-login plugin with the below Krew command:

kubectl krew install kubectl-login

Your OpenID Connect provider must have the default callback endpoint for the Kubernetes API Client listed within the OpenID configuration:

http://localhost:33768/auth/callback

This Kubectl plugin takes the OpenID Connect (OIDC) issuer URL from your .kube/config, so it must be placed in your .kube/config. Once you have made this change to the kubeconfig file, you can proceed to use your username assigned to your OIDC provider:

kubectl login nigeldouglas-oidc

After this command is executed in your CLI, the browser will be opened with a redirect to the OpenID Connect Provider login page. The tokens in your kubeconfig file will be replaced after a successful authentication on the OIDC provider’s end.

7. Kubectl-whisper-secret

Link to GitHub Repository

We mentioned the importance of securing sensitive credentials like ‘Secrets’ using the kubectl-ssm-secret plugin. The whisper-secret plugin focuses on creating those secrets with improved privacy. The plugin allows users to create secrets with secure input prompts to prevent information leakages through terminal (bash) history, shoulder surfing attacks, etc.

You can install the whisper-secret plugin with the below Krew command:

kubectl krew install whisper-secret

’kubectl create secret’ has a few sub-commands we use most often that can possibly leak sensitive information in multiple ways, as mentioned above. For example, you can connect to a Docker registry via the ’kubectl create secret’ command with a plain-text password for authentication.

kubectl create secret docker-registry my-secret --docker-password nigelDouglasP@ssw0rD

’kubectl whisper-secret’ plugin allows users to create secrets with a secure input prompt for fields like –from-literal and –docker-password that contain sensitive information.

kubectl whisper-secret docker-registry my-secret --docker-password -- -n nigel-test --docker-username <insert-password>

You are then prompted to enter the Docker password, but this is not inserted into the command itself. This way, the password will not show-up in the bash history as a plain text value, increasing security.

8. Kubectl-capture

Link to GitHub Repository

Sysdig open source (Sysdig Inspect) is a powerful tool for container troubleshooting, performance tuning, and security investigation. The team at Sysdig created a kubectl plugin which triggers a packet capture in the underlying host which is running a pod.

You can install the kubectl-capture plugin with the below Krew command:

kubectl krew install kubectl-capture

Packet captures are incredibly useful for incident response and forensics in Kubernetes. The capture file is created for a duration of time and is downloaded locally in order to use it with Sysdig Inspect, a powerful open source interface designed to intuitively navigate the data-dense Sysdig captures that contain granular system, network, and application activity of a Linux system.

Simply run the below command against any running pod in the cluster:

kubectl capture kinsing-78f5d695bd-bcbd8

When the capture container is being spun, it takes some time to compile the Sysdig Kernel module and capture system calls. Once completed, you can read the content within the Sysdig Inspect UI from your workstation:

alt_text
With these tools, it will be much easier for the analyst to find the source of the problem or to audit what happened. If you want to go deeper, you can read container troubleshooting with Sysdig Inspect or triaging malicious containers.

9. Kubectl-trace

Link to GitHub Repository

kubectl-trace is a kubectl plugin that allows you to schedule the execution of bpftrace programs in your Kubernetes cluster. In short, Kubectl-trace plugin is a tool for distributed tracing in Kubernetes clusters. It allows you to trace the execution of requests as they pass through different components of a cluster, including pods, services, and ingress controllers.

You can install the kubectl-trace plugin with the below Krew command:

kubectl krew install trace

One potential security benefit of using the Kubectl-trace plugin is that it can help you identify and troubleshoot issues related to request handling within a cluster. For example, if you suspect that a particular request is being blocked or slowed down due to some issue in the cluster, you can use Kubectl-trace to track the request as it travels through the cluster and identify the source of the problem.

This plugin runs a program that probes a tracepoint on the node of choice:

kubectl trace run <node-name> -e "tracepoint:syscalls:sys_enter_* { @[probe] = count(); }"

Another potential security benefit is that Kubectl-trace can help you understand how requests are being handled within a cluster, which can be useful for identifying potential vulnerabilities or misconfigurations. For example, if you see that a request is being handled by a pod or service that has been compromised, you can use Kubectl-trace to track the request and identify the source of the issue.

Overall, the Kubectl-trace plugin can be a useful tool for improving the security of a Kubernetes cluster by helping to identify and address issues related to request handling and execution.

10. Access-matrix

Link to GitHub Repository

Access-matrix (often referred to as ‘Rakkess’) is a kubectl plugin that shows an access matrix for your server resources.

You can install the access-matrix plugin with the below Krew command:

kubectl krew install access-matrix

Simply run the below command to see the Create, Read, Update & Delete (CRUD) permissions for all resources in the ‘default’ network namespace:

kubectl rakkess –n default

Some roles only apply to resources with a specific name. To review such configurations, provide the resource name as an additional argument. For example, show access rights for the ConfigMap called sysdig-controller in namespace sysdig-agent:

kubectl access-matrix r cm sysdig-controller -n sysdig-agent --verbs=all

As rakkess resource needs to query Roles, ClusterRoles, and their bindings, it usually requires administrative cluster access.

11. Rolesum

Link to GitHub Repository

The Rolesum kubectl plugin is a tool for generating a summary of the roles and permissions defined in a Kubernetes cluster. It allows you to see all of the roles and permissions that have been defined in a cluster, along with the users and groups that have been granted those roles. Summarize RBAC roles for the specified subject (ServiceAccount, User, and Group).

You can install the rolesum plugin with the below Krew command:

kubectl krew install rolesum

One potential security benefit of using the Rolesum kubectl plugin is that it can help you identify and understand the roles and permissions that have been defined in a cluster. This can be useful for ensuring that appropriate access controls have been put in place, and for identifying potential vulnerabilities or misconfigurations.

You can summarize roles bound to the “nigeldouglas” ServiceAccount.
By default, rolesum looks for serviceaccounts. There’s no need to specify any flag.

kubectl rolesum nigeldouglas

Another potential security benefit is that Rolesum can help you quickly identify users and groups that have been granted certain roles or permissions, which can be useful for troubleshooting issues or for performing security assessments.

For example, you can summarize roles bound to the “staging” group.

kubectl rolesum -k Group staging

Overall, the Rolesum kubectl plugin can be a useful tool for improving the security of a Kubernetes cluster by helping you understand and manage the roles and permissions that have been defined in the cluster.

12. Cert-Manager

Link to GitHub Repository

Cert-Manager is a Kubectl plugin that provides automatic management of Transport Layer Security (TLS) certificates within a cluster. It allows you to easily provision, manage, and renew TLS certificates for your applications without having to manually handle the certificate signing process.

You can install the cert-manager plugin with the below Krew command:

kubectl krew install cert-manager

One potential security benefit of using cert-manager is that it can help you ensure that your applications are using valid, up-to-date TLS certificates. This can be important for protecting the confidentiality and integrity of communication between your applications and their users.

Another potential security benefit is that cert-manager can help you automate the process of obtaining and renewing TLS certificates, which can reduce the risk of certificate expiration or mismanagement.

Overall, the cert-manager kubectl plugin can be a useful tool for improving the security of a Kubernetes cluster by helping you to manage TLS certificates in a secure and automated manner. The cert-manager plugin is loosely based upon the work of kube-lego and has borrowed some wisdom from other similar projects, such as kube-cert-manager.

13. np-viewer

Link to GitHub Repository

The kubectl-np-viewer plugin is a tool for visualizing the network topology of a Kubernetes cluster. It allows you to view the connections between pods, services, and other resources within a cluster in a graphical format.

You can install the np-viewer plugin with the below Krew command:

kubectl krew install np-viewer

Unlike the Cilium plugin we mentioned earlier, the kubectl-np-viewer plugin helps users understand and visualize the communication patterns within a cluster regardless of the CNI plugin used. The Cilium plugin only helps manage Cilium resources, such as the Cilium network policy. By viewing the default Kubernetes network policies, teams who are starting off with Kubernetes networking benefit from useful visibility into potential vulnerabilities or misconfigurations, such as pods that are communicating with unintended resources or are exposed to the internet.

The below command prints network policies rules affecting a specific pod in the current namespace:

kubectl np-viewer -p pod-name

Similarly, a potential security benefit from the kubectl-np-viewer plugin is that it helps users troubleshoot network issues within a cluster. For example, if you are experiencing connectivity issues between pods or services, you can use the plugin to visualize the connections between those resources and identify the source of the problem across all network namespace.

The below command prints all network policies rules for all namespaces:

kubectl np-viewer --all-namespaces

Overall, the kubectl-np-viewer plugin can be a useful tool for improving the security of a Kubernetes cluster by helping you to understand and monitor the network topology of the cluster. Not all businesses have moved to advanced network policy implementations, such as Calico and Cilium. While users are exploring the Kubernetes Network Policy implementation, they can better understand how their policies control potentially unwanted/malicious traffic within their cluster with this security plugin.

14. ksniff

Link to GitHub Repository

The ksniff kubectl plugin is a tool for capturing and analyzing network traffic in a Kubernetes cluster. It can be used to troubleshoot network issues, monitor traffic patterns, and perform security assessments.

You can install the ksniff plugin with the below Krew command:

kubectl krew install ksniff

One benefit of using ksniff is that it allows you to capture and analyze traffic without having to directly access the nodes in a cluster. This can be helpful in situations where you don’t have direct access to the nodes, or where you want to minimize the potential impact of capturing traffic on the cluster.

Another benefit is that ksniff can be used to capture traffic between pods and services, which can be useful for understanding how applications communicate within a cluster. This is helpful for troubleshooting issues, optimizing performance, and identifying potential security vulnerabilities.

Overall, the ksniff kubectl plugin can be a useful tool for improving the security of a Kubernetes cluster by helping to identify and address network-related issues and vulnerabilities. It achieves this by sniffing on Kubernetes pods with existing technologies, such as TCPdump and WireShark.

15. Inspektor-Gadget

Link to GitHub Repository

Inspektor-Gadget is one of the most useful kubectl plugins. The plugin executes within the user’s system and as a DaemonSet when deployed within the cluster. It is actually a collection of tools (or gadgets) to debug and inspect Kubernetes resources and applications.

You can install the gadget plugin with the belowKkrew command:

kubectl krew install gadget

You can deploy one or more gadgets. Example gadgets are categorized into:

It manages the packaging, deployment, and execution of eBPF programs in a Kubernetes cluster, including many based on BPF Compiler Collection (BCC) tools, as well as some developed specifically for use in Inspektor Gadget. It automatically maps low-level kernel primitives to high-level Kubernetes resources, making it easier and quicker to find the relevant information.

To “Advise” on a Kubernetes Network Policy based on network trace activity, run the below command:

kubectl gadget advise network-policy report --input ./networktrace.log > network-policy.yaml

To “Audit” a seccomp profile based on pods, namespaces, syscalls, and code, run the below command:

kubectl gadget audit seccomp -o custom-columns=namespace,pod,syscall,code

DIY kubectl plugins

You can write a plugin in any programming language or script that allows you to write command-line commands. There is no plugin installation or pre-loading required, which makes compiling these plugins rather simple.

Plugin executables receive the inherited environment from the kubectl binary.

The plugin will then determine which command path it wishes to implement based on the name – for example, a plugin named kubectl-sysdig provides a command kubectl sysdig.

You must install the plugin executable somewhere in your PATH.

A plugin script would look something like this:

#!/bin/bash
# optional argument handling
if [[ "$1" == "version" ]]
then
    echo "1.0.0"
    exit 0
fi
# optional argument handling
if [[ "$1" == "config" ]]
then
    echo "$KUBECONFIG"
    exit 0
fi
echo "I am a plugin named kubectl-sysdig"

For a complete guide on building Kubectl plugins, check out the official Kubernetes documentation.

Final considerations on kubectl plugins

At the time of writing this blog post, there were 208* kubectl plugins currently accessible on Krew. Those kubectl plugins are accessible to developers across all major platforms, like MacOS, Linux, and Windows. While these plugins often address clear limitations over the default kubectl utility for operational tasks and security auditing, they also open a bunch of new security gaps for your Kubernetes cluster.

From a security standpoint, we discussed 15 of the most useful kubectl plugins for giving security teams better visibility for incident response and forensics in Kubernetes.

However, as we add more plugins into the environment, we are also adding additional un-audited binaries that could be compromised. Krew does not provide an obligation to audit these binaries for known vulnerabilities or insecure configurations.

Some security implications of using kubectl plugins include:

  1. Plugin vulnerabilities: If a kubectl plugin has a vulnerability, it can potentially be exploited by an attacker to gain access to your Kubernetes cluster.
  2. Insecure plugin installation: If a plugin is installed from an untrusted source, it could contain malicious code that could compromise the security of your cluster.
  3. Privilege escalation: kubectl plugins run with the same privileges as the kubectl command, so if a plugin is compromised, it could potentially be used to escalate privileges and gain access to sensitive resources in your cluster.
  4. Data leakage: If a kubectl plugin is not properly secured, it could potentially leak sensitive data from your cluster.

To mitigate these risks, it is important to only install kubectl plugins from trusted sources. It is also a good idea to regularly review the plugins you have installed and update, patch, or remove any that are no longer needed.

If you enjoyed this content and would like to learn more about Kubernetes security, feel free to discover more at Sysdig’s ‘Learn Cloud Native’ feed for further news and updates.

Subscribe and get the latest updates