Automated Testing with Azure DevOps: Tutorial

With the rise of cloud computing, testing has rapidly moved to the cloud as well. Today, many cloud providers offer test automation solutions as scalable, on-demand services.

Azure, AWS, GCP, Oracle Cloud services, and many more cloud service providers have integrations with multiple open-source and enterprise test automation tools. Selenium is the most used framework that has integrations into all the major cloud platforms. Also, there are test platforms like BrowserStack, which offer test automation tools as SaaS (Software-as-a-Service) solutions.

This article aims at getting people started with running Automated Testing on Microsoft’s Azure. It highlights what Azure DevOps is, and then understands how to run automated tests with Azure DevOps.

What is Microsoft Azure and Azure DevOps?

Microsoft Azure, previously called Windows Azure, is the cloud computing platform by Microsoft. It is a huge collection of servers and networking hardware that hosts a complex set of distributed applications. These applications are exposed as services and range from cloud computing, analytics, networking, and storage solutions.

Azure DevOps is the service of Azure that helps users set up CI/CD pipelines for their software projects. It provides various developer services that help teams to plan work, collaborate on code development, and build and deploy applications.

Benefits of Azure DevOps Automation Testing

Here are the Benefits of Automation Testing with Azure DevOps, written as concise bullets:

• Seamless CI/CD Integration: Automates test execution within build and release pipelines.
• Faster Feedback Loops: Instantly notifies teams of test failures, improving development speed.
• Scalable Test Execution: Run tests in parallel across multiple agents or environments.
• Centralized Test Management: Track test plans, results, and defects in one unified platform.
• Supports Multiple Test Frameworks: Works with Selenium, NUnit, JUnit, Appium, and more.
• Integration with Cloud Browsers and Devices: Easily connect with tools like BrowserStack for real device testing.
• Customizable Pipelines: Define workflows with YAML or GUI to suit project needs.
• Enhanced Collaboration: Devs, testers, and ops can work together in one DevOps ecosystem.

Introduction to Azure Mobile Apps

Azure Mobile Apps helps developers and system integrators with a mobile application development platform. The platform is highly scalable and provides your mobile application with authentication, data query, and offline data synchronization.

Using the Azure Mobile apps platform you can:

• Build native apps
• Build cross platform apps
• Authenticate and connect users to your enterprise data stores
• Build offline apps with data sync

Run Selenium Tests on Azure DevOps

To run a Selenium test suite on Azure DevOps, listed below are some of the pre-requisites that you will need:

• Azure DevOps account: Free account can be created
• Knowledge of Selenium, Maven, Java, and TestNG
• A working test suite that uses the above technologies
• Github repo to store the test suite code

Step 1: Create an Azure DevOps account (If you already have an account, proceed to step 2)

Account can either be created using your email id or if you already have a Github account, Azure DevOps account can be created with the Github account. Once an account is created, sign in to Azure DevOps.

Step 2: Create a new Project

Once you log in to your account, click on Create Project. This will open a dialogue box where you can fill in the project name, a description and also set the visibility (public/private) of the project.

Once the project is created, you will land on the project home page. This will have options to create pipelines, setup dashboards, test plans, wiki pages, and tracking boards.

Step 3: Create a Pipeline

As a next step, we will create a pipeline that we will link with our test repo and issue commands to perform the tests.

Step 4: Link your code repo

The test automation code repo needs to be stored in a version control system and in this step, we will need to integrate that into the pipeline. Azure offers integration with Azure Repos, Bitbucket Cloud, Github, and Subversion. For this demo, we will go ahead with the Github option.

Once you have linked your Github account with the Azure DevOps account, all the repos in your organization start to appear on the Select a repository page.

Talk to an Expert

Step 5: Select the task

For this demo, let us select the Maven task that will help us build java projects and run tests with Apache Maven. Upon selecting Maven, the review page will appear. Click on Save and run.

Step 6: Configure Maven goals

Configure the Maven goals. Add any options you would wish to pass to maven. Also, pass the test report location.

Step 7: Pipeline run

Click on Run. This will open a dialogue box as below. Select the agent pool, agent, and the branch from which Azure DevOps needs to execute the build. A commit ID can also be given to build a specific commit. On leaving this field empty, the latest commit on the branch shall be built.

Check the Enable system diagnostics checkbox and click Run. Once the pipeline is triggered, it will queue the build and wait for the executor to be available. Once we have the executor, the build starts, and the progress can be viewed by clicking on the Job.

The job progress can be viewed by clicking on the jobs section.

This will run the pipeline and display the results on the same page. Cumulative results are also shown.

Integrating Browserstack with Azure Pipelines

Azure Pipelines is a Microsoft product. It is a Continuous Integration (CI) server, which automates the build and deploys process of your native web apps. There is an easy way to integrate BrowserStack Automate to run Selenium Tests with Azure Pipelines.

By executing your Appium test suite in Azure Pipelines, you can automate the testing of Mobile Applications as a part of the build process. It enables continuous testing, build, and deployment of iterative code changes. CI/CD tools help in catching failures ahead of the production stage and mitigate them as they occur. Adding BrowserStack to the mix, your native apps can be tested across 3500+ real devices.

BrowserStack lets you integrate your Appium tests with Azure Pipelines using an extension that connects the CI server to the BrowserStack Appium Grid.

Start running tests on 3500+ real browsers and devices on BrowserStack’s real device cloud. Run parallel tests on a Cloud Selenium Grid to get faster results without compromising on accuracy. It helps in detecting bugs before users do it by testing software in real user conditions with BrowserStack Automate.

Features of BrowserStack Automate:

• Executes hundreds of concurrent tests
• Integrates with popular languages like Python, Java, and top CI/CD tools like Jenkins, CircleCI, etc.
• Offers instant access to 3500+ Real Devices and Browsers
• Comprehensive Debugging using video recordings, automated screenshots of errors
• Enterprise-Grade Security & GDPR Compliance

Best Practices for Test Automation in Azure DevOps

Here are best practices for Test Automation in Azure DevOps, written as clear and actionable one-line bullets:

• Use YAML pipelines for version control: Keep your pipeline definitions in source control for transparency and reusability.
• Run tests in parallel across agents: Speed up execution by distributing tests using Azure’s multi-agent support.
• Use stages to separate build, test, and release: Isolate each step for better organization and debugging.
• Integrate with test reporting tools: Use built-in test results publishing or tools like ReportPortal for insights.
• Store secrets securely with Azure Key Vault: Never hardcode credentials or tokens in your pipeline.
• Use conditionals to run relevant tests: Trigger specific tests based on branch, file changes, or environment.
• Automatically capture test artifacts: Archive logs, screenshots, and reports for failed tests to aid debugging.
• Monitor flaky tests and tag them: Isolate unreliable tests to avoid blocking deployments.
• Integrate with external test grids like BrowserStack: Extend test coverage with real browsers and devices.
• Use variables and templates: Simplify pipeline reuse across multiple projects or environments.

Conclusion

Automating tests within Azure DevOps pipelines helps teams catch issues early, speed up release cycles, and build with confidence. But testing on just one browser or device isn’t enough.

By integrating BrowserStack Automate, you can run your Selenium or Appium tests across thousands of real browsers and devices in the cloud—no setup, no maintenance. It’s a simple way to make your test coverage more complete and your releases more reliable.

Try BrowserStack for Free