DataOps for Power BI

& Microsoft Fabric

2023-09-13
 01 02
DevOps & DataOps Concepts Power BI Deployment

CONTENTS
Pipelines

03 04
Power BI & Azure DevOps Power BI Desktop projects
GIT integration

05 06
Lifecycle management Roadmap
DevOps & DataOps Concepts
DevOps is the union of people, Develop
+ Test

processes, and products to Plan +

Track
enable continuous delivery of Continuous
Delivery

value to our customers Release

Monitor +
Operate

Shorten cycle Quality Optimize Continuous

times improvement resources innovation thru
and deliver value thru continuous and eliminate experimentation
faster feedback waste
DevOps
Develop

DevOps is the union of people,

process, and products to enable
continuous delivery of value to
your end users

DevOps
Cross functional teams

Operate Deliver
What is DataOps
DataOps is a collaborative data management
practice focused on improving the communication,
integration and automation of data flows between
data managers and data consumers across an
organization

DataOps checklist - Azure Architecture Center

Power BI Deployment
Pipelines
Deployment pipelines

Enable Power BI creators

build an efficient &
reusable process to deploy
new content to users.
DEMO 1 – Power BI Deployment Pipelines
What attributes are deployed?
Supported items Unsupported items

• Datasets • Datasets that don't originate from a PBIX

• Reports • PUSH datasets
• Dataflows • Streaming dataflows
• Datamarts • Reports based on unsupported datasets
• Dashboards • Template app workspaces
• Paginated reports • Workbooks
• Metrics

Copy & override No copy, no override Copy with rules

• Report visuals • Data • Datasources

• Report pages • URL & ID • Parameters
• Dashboard tiles • Permissions • More will be available in the future
• Model metadata • WS settings
• Item relationships • App content and settings
• Refresh schedule
 Deployment pipeline permissions
• View pipeline
• Share with others
• Edit & delete pipeline
• View the pipeline • Consume content • Consume content • View workspace • View workspace

Workspace admin
Workspace viewer

Workspace member
Pipeline admin

(and pipeline admin)

(and pipeline admin)

(and pipeline admin)

(and pipeline admin)

Workspace contributor
• Share the pipeline • Unassign a • Compare stages content content
with others workspace from a • View datasets • View WS content
• Compare stages • Compare stages
• Edit and delete stage • Unassign a • Compare
• Deploy stages
items • Deploy items
the pipeline workspace from a • (mustreports
Deploy be a & dashboards • Assign
• Unassign a stage member or admin workspaces to a
workspace from a of both source stage
stage and target • Update datasets
workspaces)
• Can see • Assign/ remove WS • Unassign a
workspaces that • Update datasets workspace from a
are tagged as • Unassign a stage
assigned to the workspace from a • Configure dataset
pipeline in Power stage rules (you must be
BI service • Update dataset
• Configure dataset the dataset
• rules (you
Configure must be
rules owner)
the dataset
owner)

Deployment pipelines, the Power BI Application lifecycle management (ALM) tool, process
Best practices for deployment pipelines, the Power BI Application lifecycle management (ALM) tool
Automate Deployment Pipelines with APIs
With the Power BI Rest APIs, you can integrate Power BI
into a comprehensive automation process. This includes:

• Pipeline Provisioning and Management

• User Access Management for Pipeline Leverage the Deployment Pipelines APIs to perform
• Schedule Pipeline Deployment functions such as:
• Deploy Pipelines with Cascading Dependencies • Checking deployment status & history
• Creating and Deleting Pipelines
• Assigning and Unassigning workspaces to stages
There are several APIs can inject automation into to
• Add and Remove Pipeline Users
enable a more efficient deployment depending on your
developers’ requirements:

Deploy All Selectively deploy Backward deploy new Update content

Workpace specific Power BI Power BI items to of app for the
Content assets previous stage stage
Power BI & Azure DevOps
Introducing Azure DevOps

Azure Boards Azure Pipelines Azure Repos

Deliver value to your users faster Build, test, and deploy with CI/CD that Get unlimited, cloud-hosted
using proven agile tools to plan, works with any language, platform, private Git repos and collaborate
track, and discuss work across and cloud. Connect to GitHub or any to build better code with pull
your teams. other Git provider and deploy requests and advanced file
continuously. management.

Azure Test Plans Azure Artifacts

Test and ship with confidence Create, host, and share packages with ➔
using manual and exploratory your team, and add artifacts to your
testing tools. CI/CD pipelines with a single click. https://azure.com/devops
Automate Deployment Pipelines with Azure DevOps
Automating Deployment Pipelines enables agility across
your organization’s CI/CD approach by applying
techniques such as scheduled deployments, reusable
releases to your Power BI content management Power BI Automation Tools are an open-source
alternative to APIs and Scripts with individual
There are 2 core methods to achieve this: operations such as create a pipeline, assign
• Azure DevOps Integration workspace, etc.
• Power BI Rest APIs

With Azure DevOps, PowerShell or Power BI Automation

Tools can be used to facilitate the automation tasks.
• Power BI Automation Tools use a service connection to
Azure DevOps is established with a service principal or Leverage the Power BI Developer Samples for
username & password PowerShell Scripts to automate selective
deployment, end to end backwards deployment, etc.
• PowerShell uses a service principal or user through the script
to facilitate the deployment and get notified about its
completion status
Azure Pipelines Marketplace

Power BI automation tools | Power BI Actions

DEMO 2 – Azure DevOps & Power BI Automation Tools
What is your org’s governance approach?
A formalized checklist to clarify standards that the Power BI content must meet in order
to be approved and promoted to Production
Decision/Action:
Approval Criteria & ✓ Agree on and document your approval criteria which may include compliance with
Process Org Development Standards, Data Policies, etc.
✓ Establish the intake process to request CI/CD approval via your IT Service
Management Tool and communicate to Power BI creators.

For many organizations, the approvers may be Power BI Workspace Admins. However,
in some cases approvers may include stakeholders from the Power BI, DevOps, and Data
Governance Team to provide a holistic approach.
Decision/Action:
CI/CD Approvals ✓ Ensure approvers and developers understand the approval guidelines through
training and reinforcement
✓ Incorporate CI/CD FAQs and knowledge articles into knowledge base with an
identified owner to update knowledge periodically.
 What is your org’s governance approach?
Access to Power BI reports or deployment pipelines can be managed across both the
CI/CD pipeline and Power BI Portal when automating deployments
• Required API permissions for the CI/CD Pipeline service account:
• Create/manage workspaces, Publish / Deploy / Edit artifacts.
• Required permissions for the CI/CD Pipeline developer:
• Create/Manage Azure DevOps organization and Projects.
• Access to Azure Key Vault
Power BI Content and • Workspace Member permissions that allow managing Datasets, Dataflows and other
Access Management artifacts

Decision/Action:
✓ Extend pipeline and workspace permissions to the service principal or users to
provide enough access & resilience to facilitate deployment across Dev-Test-Prod
✓ Decide if all deployments should be routed through CI/CD, and if Power BI
deployments made via Power BI Deployment Pipelines page will be unauthorized.
 Roles & Responsibilities: CI/CD Key Roles

Power BI Developer Power BI Content Approver Power BI Deployment Pipeline

(CI/CD Requestor) (CI/CD Approver) Admin (CI/CD Pipeline Admin)

Role Quote: Role Quote: Role Quote:

“I develop Power BI content that is “I manages approvals of Power BI “I ensure that deployments are performed
managed by continuous updates to meet deployments based on our compliance in an efficient and automated way.”
Business requirements.” requirements.”
Skills & Training: Skills & Training:
Skills & Training:
Monitor performance of Power BI Define & Adapt governance for CI/CD
content and make changes as needed Perform Deployments via Azure DevOps
deployments
throughout its lifecycle & Troubleshoot Pipeline functions
Understand and enforce org guidelines
Develop Power BI Content following data Establish CI/CD environment including
on Power BI data & development
& development standards workspaces, access, etc.
standards
Managing Deployment Operations

Development Test Production

• Workspace Admin • Workspace Admin • Workspace Admin
• Dataset Owner • Dataset Owner • Dataset Owner
• Pipeline Access (View, Share, • Pipeline Access • Pipeline Access
Edit & Delete Pipeline)

• Workspace Viewer • Workspace Viewer • Workspace Viewer

• Workspace contributor • Workspace Viewer • Workspace Viewer

• Git Repo Access

Decision/Action: How do you plan to manage access across each stage?

High-Level CI/CD and Power BI Approval Workflow

24
DEMO 3 – Customizable DevOps Pipeline with PowerShell
Open semantic models – XMLA Endpoint

Read-only capability for the XMLA endpoint Excel

provides open-platform connectivity for Power BI
datasets compatible with a range of data
visualization tools from different vendors
Security

In-memory Data
cache modeling

Text
Read/write capabilities support many additional Power BI reports
Power BI
SQL Server
& paginated Management
scenarios for dataset management, advanced reports datasets Studio
semantic modeling, debugging, and monitoring
Business logic
ALM & metrics

With XMLA Endpoint, you can refresh more than Third party
48 / days for incremental refresh, push data, …
SQL Server
Data Tools
 Advanced Data Model Management

Power BI usage scenarios: Advanced data model

 Data Model metadata as a Folder
• Save model metadata as a
folder structure for better
version control integration

• Command line tool for build

and deployment automation

• Improve model quality with the

Best Practice Analyzer

• Workspace mode for improved

development workflow

Tabular Editor 2 - Tabular Editor 3 Automate Power BI Best Practice Analyzer

 Compare Data Model

ALM Toolkit
DEMO 4 – Advanced DevOps Pipeline with DAX & BPA
Highlighting data model changes easily
 Best Practices
• Get Version Control

• Separate Reports From the Model (ETL – Data Model – Report)

• Test:
• The presence of a value in a column that shouldn't be there
• Number of rows in a fact table / dimension
• The existing of blank columns
• The values in a custom column are correct

• Have a Test Dataset (Only code change)

• Institute Workspace Governance

• No Data within Code Repository

• It’s possible to Backup Power BI Dataset

• Monitor & Foster a Power BI Center of Excellence

Power BI Desktop projects
Git integration
Power BI Project
Co-development with Azure DevOps
DEMO 5 – Power BI Desktop developer mode

Power BI Desktop projects (PBIP)

Lifecycle management
Power BI usage scenarios: Enterprise content publishing
Continuous Integration with Fabric Git Integration
DEMO 6 – Lifecycle management

Lifecycle management tutorial

Git integration with paginated reports

Git integration with paginated reports

 Deployment Options
Option License State Features Best for
Individual citizen
PBIX File Pro GA Manual publish, No change tracking
developers
Developers and small
Manual publish, Granular change
PBIP Project Pro Preview teams with source
tracking via GIT
control skills
Multiple environment, Simple change
Power BI Deployment Small team with source
Premium Preview tracking with compare view for tabular
Pipelines control skills
model
Multiple environments, Change
Larger teams with source
GIT Integration Premium Preview tracking, Azure DevOps only (during
control needs
Preview)
Fine grained dataset refresh,
Pro or Enterprise teams with
Custom CI/CD GA Automated setting of credentials,
Premium need for full control
Testing, Assessment, …
DataOps Principles #18
1. Continually satisfy your customer
2. Value working analytics
3. Embrace Change
4. It’s a team sport
5. Daily interactions
6. Self-organize
7. Reduce heroism
8. Reflect
9. Analytics is code
10. Orchestrate
11. Make it reproducible
12. Disposable environments
13. Simplicity
14. Analytics is Manufacturing
15. Quality is Paramount
16. Monitor for Quality and Performance
17. Reuse
18. Improve cycle times
Thank you