Microsoft has warned of a nefarious campaign that has been targeting Kubeflow. If you're unfamiliar with Kubeflow, it is a toolkit that is used to integrate machine learning (ML) features onto Kubernetes clusters.
In a report, Microsoft says the new attack on Kubeflow has never been seen before. According to the company, bad actors have been leveraging the method since April. It seems attackers want to be able to use Kubernetes to power their cryptocurrency mining operations.
Yossi Weizman, a security researcher with Microsoft's Azure Security Center, says the company has found “tens of Kubernetes clusters” that have been affected by the Kubeflow attack. That may not be a massive amount, the financial cost of these attacks could be higher than normal.
“Nodes that are used for ML tasks are often relatively powerful, and in some cases include GPUs,” Weizman points out.
“This fact makes Kubernetes clusters that are used for ML tasks a perfect target for crypto mining campaigns, which was the aim of this attack.”
Attack
Microsoft first became aware of the attacks in April and has been tracking them since. The company says the attacks have escalated over time, not including targeting machine learning clusters. Microsoft believes misconfigured Kubeflow instances give attackers a gateway into Kubernetes.
Specifically, it is believed admins in control of Kubeflow accidentlly switched default settings and opened the toolkit's admin panel. It is supposed to only be seen internally, but Microsoft thinks it may have been exposed online.
Microsoft points to two methods for checking is a cluster has been hacked:
- “Verify that the malicious container is not deployed in the cluster. The following command can help you to check it:
kubectl get pods –all-namespaces -o jsonpath=”{.items[*].spec.containers[*].image}” | grep -i ddsfdfsaadfs
- In case Kubeflow is deployed in the cluster, make sure that its dashboard isn't exposed to the internet: check the type of the Istio ingress service by the following command and make sure that it is not a load balancer with a public IP:
kubectl get service istio-ingressgateway -n istio-system”