A recently disclosed vulnerability in Kubernetes dashboard (CVE-2018-18264) exposes secrets to unauthenticated users. In this blog post we’ll explore some key takeaways regarding monitoring privilege escalation on Kubernetes.
Kubernetes dashboard CVE-2018-18264 overview
The Kubernetes dashboard is a web based user interface that allows users to manage applications and resources within the cluster. This service has a login functionality starting from Kubernetes 1.7.0. Since then, users are able to authenticate with a kubeconfig file or an access token. But it is also possible to skip the authentication altogether with a skip button.
The recently disclosed vulnerability in Kubernetes dashboard: CVE-2018-18264 allows users to escalate privileges within the Kubernetes cluster using the dashboard to potentially access information they are not actually granted access to: a user could skip the authentication and query resources that the dashboard service account has access to, like the kubernetes-dashboard-certs secret, that stores TLS certificate and private key for the service.
This is particularly critical because with those TLS credentials, the attacker could impersonate the dashboard service, spawning a fake service to steal other user’s credentials.
Kubernetes dashboard v1.10.1 was released to address this vulnerability, this is the announcement email and see PRs #3289 and #3400 for details.
Skipping authentication and using the service account credentials is an unusual behavior and this vulnerability should raise concerns on how these actions can go unnoticed by monitoring and security tools. Monitoring CVE-2018-18264 #Kubernetes dashboard vulnerability for privilege escalation #security attacks Click to tweet
Privilege escalation as seen by Kubernetes Audit
The Kubernetes Audit Policy is a new feature that allows the Kubernetes API server to generate logs regarding API calls. This allows to answer in detail the question of who requested what within the Kubernetes cluster. At first sight, It seems the right tool to chase an attacker who has retrieved a cluster secret exploiting this vulnerability.
Let’s try to reproduce the attack, and see if that’s the case.
By using an unpatched version of the Kubernetes dashboard, we’re able to hit the skip button and paste the request for a secret, in this case the kubernetes-dashboard-certs, which resides at/api/v1/namespaces/kube-system/secrets/kubernetes-dashboard-certs.
If we inspect the Kubernetes audit events logs generated in the described scenario, here’s what we would see:
{
"kind":"Event",
"apiVersion":"audit.k8s.io/v1beta1",
"metadata":{"creationTimestamp":"2019-01-08T07:31:06Z" },
"level":"RequestResponse",
"timestamp":"2019-01-08T07:31:06Z",
"auditID":"8e4b1f58-55db-4c80-be33-fc182637783c",
"stage":"ResponseComplete",
"requestURI":"/api/v1/namespaces/kube-system/secrets/kubernetes-dashboard-certs",
"verb":"get",
"user":{
"username":"system:serviceaccount:kube-system:default",
"uid":"ce751d56-1313-11e9-b856-08002717c1f6",
"groups":[
"system:serviceaccounts", "system:serviceaccounts:kube-system", "system:authenticated"
]
},
"sourceIPs":["172.17.0.4"],
"objectRef":{"resource":"secrets","namespace":"kube-system","name":"kubernetes-dashboard-certs", "apiVersion":"v1"},
…
"annotations":{
"authorization.k8s.io/decision":"allow",
"authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding "minikube-rbac" of ClusterRole "cluster-admin" to ServiceAccount "default/kube-system""
}
}
Code language: JavaScript (javascript)From the Kubernetes API server, the request is coming from the cluster IP 172.17.0.4, corresponding to the Kubernetes dashboard, and the user field is populated using the default kube-system’s Service Account (i.e. system:serviceaccount:kube-system:default).
It’s clear that there’s no additional information regarding the fact that this request is forwarded from an unauthenticated session. In fact, in the current Kubernetes implementation, the Kubernetes dashboard access the resource on behalf of the user. This means that the authorization is performed by the Kubernetes dashboard, before the request is issued to the Kubernetes API server.
We can also notice that a very similar audit log will appear every time a Kubernetes dashboard container is deployed. Indeed, during the startup phase, thekubernetes-dashboard-certs is fetched and then stored in memory.
How to detect Kubernetes privilege escalation security attacks
This vulnerability has been very promptly patched, just make sure you’re running Kubernetes dashboard v1.10.1, where the skip login has also been disabled by default. Still concerns should be raised regarding the possibility that new exploits of this kind would not be easily detectable.
As matter of fact, these kind of privilege escalation exploits can be very hard to detect, as the single event is rightly authorized by the Kubernetes API server, despite being forwarded on behalf of an unauthorized user.
A good practice might be to always monitor the activities around critical resources like secrets.
Starting from release version 0.13.0, Sysdig Falco supports Kubernetes Audit Event stream. This means that you can create rules that trigger notifications whenever there’s an access to a given resource.
For instance, to detect a possible exploit of CVE-2018-18264, we could use Falco to fire an alert when there is an access to the kubernetes-dashboard-certs secret. The Falco rule would look like this:
- macro: secrets
condition: (ka.target.resource=secrets)
- rule: Access to Kubernetes Dashboard Certificate
desc: Match all access to secret kubernetes-dashboard-certs
condition: secrets and ka.uri=/api/v1/namespaces/kube-system/secrets/kubernetes-dashboard-certs
output: K8s Audit Event received (user=%ka.user.name verb=%ka.verb uri=%ka.uri obj=%jevt.obj)
priority: DEBUG
source: k8s_audit
tags: [k8s]
Code language: JavaScript (javascript)Remember that in order to make this work you also need to configure your audit policy file to generate audit events on the namespace and resources that we might be interested in, like this:
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
- level: Metadata
resources:
- group: ""
resources: ["secrets"]
resourceName: ["kubernetes-dashboard-certs"]
namespace: ["kube-system"]
Code language: JavaScript (javascript)The Kubernetes dashboard generally requests the certificate once at boot. You might be interested in auditing Kubernetes dashboard creation as well, and by correlating both events you would be able to infer that the certificate request is legitimately requested by the pod upon startup. Otherwise, if you start seeing accesses to the kubernetes-dashboard-certs without any other events you can relate, you might want to perform some troubleshooting in search for privilege escalations.
Although given the current resource and identity access management, it is not possible to determine deterministically whether a credential retrieval is due to the CVE exploit, runtime security can identify and alert privilege escalation attempts like the one we have just seen.
Kubernetes privilege escalation attacks can be also detected with Sysdig Secure. Leveraging Falco rule language, you can define runtime security policies and apply them to different Kubernetes scopes, either using the UI or as code.
Conclusions
Detecting privilege escalation exploits like this one can be very hard because an event which is authorized by the Kubernetes API server but actually comes from an unauthorized user.
Making use of the Kubernetes Audit functionality, Falco and Sysdig Secure can be used to monitor and alert on anomalous behavior and catch privilege escalation attacks and other Kubernetes security threats.