Microsoft Azure - Create SHH Key to Manage Azure Linux VMs

If you're running Linux virtual machines (VMs) on Microsoft Azure, SSH key usage is the most secure and efficient way for remote access management. SSH keys act as an electronic identification card — instead of using standard passwords, SSH relies on solid encryption which secures your servers against hackers and brute force.

With SSH key-based login, you do not need to keep remembering complex passwords or worry about phishing attacks. It is faster, more secure, and most recommended by Microsoft and leading cloud service providers. And this process also complies with enterprise-class security standards like HIPAA, GDPR, and ISO 27001, so it is ideal for personal and corporate users as well.

In this article, we’ll walk you through how to create and use SSH keys in Azure Portal to manage your Linux-based virtual machines. Whether you're a beginner, DevOps engineer, or system administrator, knowing how to set up SSH key authentication is essential for secure cloud computing.

What Are SSH Keys

SSH keys are virtual access cards to your cloud server. Instead of using a password, you utilize a key-based authentication system.

• Public Key: You put this on the Azure VM (or any server you'd like to access).
• Private Key: This stays safely on your local machine. You never share it.

When you try to connect, Azure checks whether your private key matches the public one — if they match, you're good to go. It's more of a lock-and-key deal. Azure VM = Lock (with your public key). Your Computer = Key (your private key)

For more details refer What are SSH Keys?

Why Use SSH Keys in Azure

We use SSH keys in azure because:

• Enhanced Security: SSH keys are much harder to crack compared to passwords. In difference to passwords, which can be guessed or stolen through phishing, brute-force attacks, or keylogging, SSH keys utilize strong cryptographic algorithms.
• Faster logins: Log into your Azure VM without entering your password every time and also automate scripts and tasks with Ansible, Terraform, or Azure CLI — without being prompted for login.
• Meets Enterprise and Compliance Standards: Many organizations — especially finance, healthcare, or government ones — require key-based authentication for remote access to all servers due to compliance standards (e.g., HIPAA, GDPR, or ISO 27001).

Step-by-Step: Create an SSH Key in Azure Portal

Follow the steps to create an SSH key in the Azure portal.

Step 1: Log in to Azure Portal

Go to Azure Portal and and sign in with your Azure credentials.

Step 2: Search for SSH Keys

In the top search bar, type “SSH Keys” and select the SSH Keys service from the results

Step 3: Configure Basic Details

Select a subscription that you manage resources and costs.

• Create a Resource Group
  • New: Create a group (e.g., “SSH-Key-Resources”).
  • Existing: Use a group you already manage.
• Region: The Azure region will auto-match your resource group.
• Key Pair Name: Give a unique name (e.g., “my-linux-vm-key”)
• SSH Public Key Source: Select “Generate new key pair”

Step 4: Add Tags (Optional)

Tags help organize resources (e.g., “Environment: Production” or “Project: WebApp”). Skip this if you’re a beginner.

Step 5: Review + Create

Now Double-check your settings.

• Click Create to generate the SSH key pair.
• Wait 1-2 minutes for Azure to create the resource.

Step 6: Download the Private Key

After creation, go to the SSH Keys resource in Azure.Under Overview, click Download private key to save the .pem file (e.g., my-linux-vm-key.pem).

How to Use SSH Key with an Azure Linux Virtual Machine (VM)

SSH key login to a Linux VM in Azure is a safe and highly-recommended practice over passwords. Not only does it secure your virtual machine from brute-force attacks, but it also offers a quicker and more automatic login process

Step 1: Create a Linux VM on Azure with SSH Key Authentication

First we need to create a Linux Virtual Machine in Azure with SSH enabled so that we can use SSH keys. Here's how:

• Go to the Azure Portal and click on the “Create a resource” than Virtual Machine.
• Under the “Basics” tab, fill in the information like:
  • Resource Group
  • VM name
  • Region
  • Image: Choose your preferred Linux distribution (Ubuntu, Debian, etc.)
• After that go under the "Administrator Account":
  • Choose the Authentication type and select the "SSH public key"
  • Username: Enter your desired VM username (e.g., azureuser)
  • SSH Public Key: Paste your generated public key here (from your ~/.ssh/id_rsa.pub file or Azure's SSH key resource)

Step 2: Connect to the Azure Linux VM Using SSH

After the VM is deployed azure assigns a public IP address. Now use your private key to connect.

• Open your Linux Terminal
• Run the SSH command

ssh -i /path/to/private-key.pem azureuser@your-vm-ip

• /path/to/private-key.pem → path to your private key (~/.ssh/id_rsa)
• azureuser → your Azure VM username
• your-vm-ip → your VM’s public IP address

Conclusion

Using SSH keys to administer Linux VMs in Microsoft Azure isn't merely a good idea — it's necessary for security, performance, and scalability. By creating a public-private key pair and setting up your Azure VMs using this method, you rid yourself of password vulnerabilities and automate your workflow.

No matter whether you're automating deployments, securely running DevOps pipelines, or simply logging into your virtual machine to work with files, SSH key authentication protects your connection as being encrypted, tamper-proof, and trustworthy.