How to remotely connect to a linux based Spark Cluster in Azure

Hi All,

In this blog post i am showing how to connect remotely to a Linux based Spark cluster in Azure.

Today, Microsoft has announced Spark general availability in Azure, read the official announcement here. Technical announcement from SQL Server team is here.

Spark GA in Azure

Once you provision a Linux based Spark Cluster, you are going to need to remotely login to it using SSH to start executing Spark commands using Spark Shell.

Open Azure Portal, Search for your cluster or you can find Spark clusters under HDInsight clusters tab; if you don't have it add it as a favorite tab from browse button in the portal).


Click on Secure Shell button, this will open a new blade with a host name that we will use to sign in using SSH to Spark cluster and start using Spark Shell.

Secure SSH to Spark from Azure Portal

Copy the host name if you are using windows and then run PuTTY tool to connect to Spark cluster using host name, username and password you have set when you provisioned the cluster.

SSH host name config

Then, Open PuTTY tool and enter the host name and then click on Open button.

Then, PuTTY will prompt to enter your username and password, once you are logged in successfully, you are in Spark Shell to start working with Spark!.

Hope this helps.