Attackers recently hijacked powerful machine-learning clusters inside Microsoft’s Azure cloud-computing service so that they could mine cryptocurrency at the expense of the customers who rented them, the company said Wednesday.
The nodes, which were misconfigured by customers, made the perfect target for so-called cryptojacking schemes. Machine-learning tasks typically require vast amounts of computing resources. By redirecting them to perform the compute-intensive workloads required to mine digital coins, the attackers found a means to generate large amounts of currency at little or no cost.
The infected clusters were running Kubeflow, an open source framework for machine-learning applications in Kubernetes, which is itself an open source platform for deploying scalable applications across large numbers of computers. Microsoft said compromised clusters it discovered numbered in the “tens.” Many of them ran an image available from a public repository, ostensibly to save users the hassle of creating one themselves. Upon further inspection, Microsoft investigators discovered it contained code that surreptitiously mined the Monero cryptocurrency.
How was it done?
After finding the infected clusters, investigators turned their attention to how the machines were compromised. For security, the dashboard that allows administrators to control Kubeflow is, by default, accessible only through istio ingress, a gateway that’s typically located at the edge of the cluster network. The default setting prevents people on the Internet at large from accessing the dashboard and making unauthorized changes to the cluster.
In a post published Wednesday, Yossi Weizman, a security-research software engineer in the Azure Security Center, said that some users change the setting.
“We believe that some users chose to do it for convenience,” Weizman wrote. “Without this action, accessing the dashboard requires tunneling through the Kubernetes API server and isn’t direct. By exposing the Service to the Internet, users can access the dashboard directly. However, this operation enables insecure access to the Kubeflow dashboard, which allows anyone to perform operations in Kubeflow, including deploying new containers in the cluster.”
Once attackers have access to the dashboard, they have multiple options for deploying backdoored containers in the cluster. For instance, attackers can create what’s known as a Jupyter Notebook server that runs on the cluster. They can then place a malicious image inside of the Jupyter Notebook. If a Jupyter Notebook is already installed, it can be maliciously modified.
Wednesday’s post provides several ways that users can check if their cluster has been compromised.
“Azure Security Center has detected multiple campaigns against Kubernetes clusters in the past that have a similar access vector: an exposed service to the Internet,” Weizman wrote. “However, this is the first time that we have identified an attack that targets Kubeflow environments specifically.”