What Is Power Query
What Is Power Query
https://gorilla.bi/power-query/beginners-guide/
https://www.ablebits.com/office-addins-blog/how-to-use-power-query-
excel/
Updated on
February 21, 2024
Power Query is an ETL (Extract Transform and Load) tool that allows you
to import data, transform it and load it into a destination.
In this beginner’s guide to Power Query, we’ll take you through the basics
of what Power Query is, how it works, and how it can help you
transform and analyze your data. By the end of this article, you’ll have a
good grasp of Power Query and how it can make your life easier. Let’s get
started!
Table of contents
Introduction to Power Query
o Power Query Capabilities
So what is Power Query in Power BI and Excel, what can it do, and what
are its benefits and limitations? Power Query is a powerful tool from
Microsoft used for transforming and preparing data. It allows you to easily
connect to and transform data from a variety of sources.
Important
With Power Query, you can make your data clean and organized with just
a few clicks. You don’t need to be a computer expert or write a lot of code.
To help you transform your data, Power Query uses an intuitive, graphical
interface. With every transformation, you see exactly what happens.
You can accomplish a wide range of data transformation tasks with Power
Query. Here are some of the things you can do with Power Query:
Connecting to data sources: Power Query makes it easy to
connect to data sources, including databases, text files,
spreadsheets, and cloud services. Once you connect to your data
source, you can preview the data, select what you want to use, and
transform it as needed.
Transforming and cleaning data: Power Query includes a range
of built-in transformation functions that allow you to clean, reshape,
and transform your data quickly. This means you can remove extra
spaces, change formats, remove duplicates and more. You can also
write custom functions using the Power Query M formula language.
Combining and appending data: Power Query lets you combine
data from different sources to make one big unified dataset. You can
merge tables based on matching columns, or append tables
vertically to create a larger data set. This even works to combine
multiple files saved within a folder.
Aggregating data: You can use Power Query to group and
summarize data, creating aggregate statistics like counts, sums,
averages, and more. This is also useful for summarizing a very large
dataset that is otherwise too big to import into Excel.
Adding columns: With Power Query, you can add new columns to
your data set, and populate them with calculated values based on
other columns in the data set. This makes it easy to create new
metrics and calculations that are based on your existing data.
In summary, Power Query can help you work with data in many different
ways. So why would you choose Power Query?
Power Query has many benefits that can make working with data much
easier. Here are some of the benefits of using Power Query:
While Power Query has many benefits, there are also some limitations to
be aware of. Here are some of the limitations:
Storing Data: Power Query does not store data physically like a
database. This means you will need to load the data into memory
each time you use it. You can output the transformed data to a
destination, but generally, a refresh will start from scratch.
Availability: Power Query is not available in every tool, which
means that you may need to use a different data transformation
tool depending on your needs. A language like SQL is supported in
many more tools compared to the Power Query M language.
Speed: Power Query can be slow when you have many queries in
your file, which can lead to frustrating wait times when working with
complex datasets.
Connectors: Power Query has a high number of native connectors
but it can’t connect to all data sources. If one is missing, you may
need to develop a custom connector.
You can find Power Query across several Microsoft products. It is currently
available in two forms.
So where can you find Power Query in Power BI and Excel (the Desktop
experience)?
In Power BI you can find Power Query through the Home tab in the
ribbon. You can get there in two ways. Either you:
To open Power Query in Excel you first navigate to the Data tab, click
Get Data and press Launch Power Query Editor.
In the last stage, you load the transformed data into a destination like:
an Excel worksheet,
the Data Model in Power BI or Power Pivot
a Dataflow in the Power BI service
a Sink in Azure Data Factory
M Formula Language
Tip
More advanced users can also write custom code manually through the
formula bar or advanced editor. This allows the user to fulfil even the most
complex requirements.
The Power Query Editor is where you do most of your data transformation
work. It consists of 7 fundamental components that help you when
transforming your data.
1. Quick Access Toolbar: A customizable toolbar at the top of the
Power Query Editor that gives access to your favourite commands.
You can save your favourite operations and access them with a
keyboard shortcut.
2. The Ribbon: A set of tabs that contains a variety of tools and
options for working with data in Power Query. The Ribbon is
organized by functional area. You will find the most important
prebuilt transformation functions in the Home, Transform and Add
Column section.
3. Query Pane: A window on the left-hand side of the editor that
displays the list of queries in the Power Query Editor. You can use
the Query Pane to organize your queries in folders, rename them,
add new ones or delete existing ones.
4. Formula Bar: A text box located at the top of the editor that
displays the M code for the selected query step. You can use the
Formula Bar to review the UI-generated formulas, learn their syntax
or edit the M code directly. In case the formula bar is invisible, make
sure to turn it on in the View tab of the Ribbon.
5. Data Preview: A grid that displays a preview of the data in the
current query. You can use the Data Preview to view and verify the
results of your transformations and ensure your data is clean and
properly formatted. You can click the column’s drop-down arrow to
rename, filter or sort your columns. Lastly, there are options to
perform actions on the entire table by clicking the Table icon in the
top-left corner.
6. Properties: Part of the Query Settings pane that displays the Query
Name.
7. Applied Steps: A list that displays the sequence of transformations
that have been applied to the data in the current query. You can use
the Applied Steps to view and modify the transformations, or to
reorder them if needed.
Power Query comes with different ribbon tabs. For your data
transformations, you will mostly use the Home, Transform, and
Add Column tabs.
Each of these three ribbons is designed with a purpose:
Ribbon Purpose
Home Tab The Home Tab contains the most frequently used
functions and provides access to data sources. You can use
this tab to create parameters, refresh your data and combine
Ribbon Purpose
tables. But also sort data, apply filters, change data types, and
join tables.
Operations in the Transform tab take place on existing
columns. Here you can perform operations such as (un)pivot
columns, split columns, and merge queries. The Transform tab
Transform
is also the place where you will find data cleaning options such
Tab
as the ability to replace values, extract specific parts of text,
and format data. Lastly you can also transform columns into a
different data type or call a Python or R script.
The Add Column tab is where you can add new columns
to your data set. It lets you create your own formulas in a
Add custom column, call a function or use AI to generate a column
Column for you. Here you can also create columns by extracting parts of
existing columns or by using context (data type) specific
operations.
To get started with Power Query, you need first to create a connection to
the data source that you want to import data from. You can connect to
databases, spreadsheets, text files, webpages, etc., as well as access
public datasets located on cloud storage services such as Google Drive
and Dropbox.
To connect to a data source you:
There are just a few options available in the drop-down. If you’re looking
for a different source, make sure to click More at the bottom. There are
hundreds of connectors hidden behind that button!
Once the connection is established, the data will land in the data preview
pane. You can then begin extracting and transforming the required data.
Transformation Examples
In this section, we’ll learn how to use Power Query to transform data.
Transforming data means changing how data looks or behaves so we can
use it in better ways. Analysts tend to spend nearly 80% of their time
cleaning data and Power Query helps do this in an effective way. Let’s
look at some examples in the below tutorial.
Sorting Data
In Power Query, you can sort your data, which means putting it in a
specific order based on certain rules. Sorting can be done in two
ways: ascending or descending. It is easy to sort your data. Simply:
Once you click Sort Ascending, your text values will be sorted
alphabetically. You can review the syntax of the generated formula in the
picture below.
Tip
If you want to sort by multiple columns, simply repeat these steps for the
next column and it will be added to the sorting formulas.
Data Types
Data types are an important part of Power Query that helps you manage
and transform your data. In Power Query, data types help you identify the
type of data you are working with, such as numbers, text, or dates. This is
useful because some operations are supported by one data type and not
by another.
There are many different data types available in Power Query. Some of
them include dates, times, text, duration, numbers, and logical values.
You can easily assign data types to columns.
In the image below, you can tell what data type a column is by looking at
the icon on the left side of the column header. If you click the icon, you
can choose to transform the column to a different data type.
By performing these steps, the columns are transformed into the data
types indicated.
Important
Filtering Data
Filtering data allows you to show only the data that meets certain criteria.
You can filter data based on specific values, ranges, or other criteria. Let’s
look at a few examples.
The most basic form of filtering is when you manually select which
columns to keep. You can click the column dropdown arrow and select the
columns you want to keep.
If you want to go further, you can use filters that meet certain criteria. The
filters provided by Power Query UI are context-dependent because
different data types require different filters.
For example, when you click on Text Filters in the above dropdown, you
can indicate the text should contain the letter “e”, which would return the
below result.
Likewise, on a number column, you can return all values between 150
and 200 by selecting the filter Between… as shown in below image:
That gives us the desired output.
Note
The following image illustrates the most common Data Type Filters:
So depending on the types your columns, you the user interface will
provide different kind of filters.
Lastly, you can also find filters for columns with the Date Type. The
picture below shows a summary of what you can do with Date Filters:
No matter which filter you choose, behind-the-scenes Power Query
translates your filters to the corresponding M code. The UI just makes sure
that these actions are within reach even to those who don’t know the M-
language formulas yet.
Transforming Columns
Merge Columns
If you have some columns with related data, you may want to combine
columns into a single column. For example, you might have a column for
First Name and one for Last Name. You can easily merge columns to
retrieve the full name.
To do that:
1. Hold CTRL and select both the First Name and Last Name columns.
2. Go to the Transform Tab in the ribbon and press Merge Columns.
3. Choose a space as a separator and name the column Full Name.
4. Now press OK.
By performing these steps, Power Query will concatenate the text values
into a single value.
Formatting Text
To get started:
Replacing Values
If you have a column with values that need to be changed, you can
replace them with new values. For example, you might have a column
with sentences that contain the word Horse. You realize you would like to
replace the text ‘horse’ with ‘pony’.
Replacing values is a simple and effective way to clean up your data and
ensure its accuracy. By following these steps, you can easily replace
values in Power Query and create a cleaner, more organized data set.
To learn about more advanced replacing scenarios, you can find step-by-
step explanations in the complete guide to replacing values.
Pivot Columns
Group or Summarize
When you press OK, your data will be summarized by the chosen column,
and all underlying values will be aggregated using the sum operation.
Tip
If you want to learn more about this topic, check out the article on
Grouping Data in Power Query. You can even write custom code to
perform additional Group By operations that are not available through the
user interface.
Adding Columns
Adding columns in Power Query is a useful feature that lets you add new
data to your data set using custom logic. You can easily create if
statements, concatenate data and create custom formulas. Let’s look at a
few examples of how you can use this feature.
Conditional Column
With an if statement you can return a value based on a condition, a so-
called conditional column. For instance, you might want to add an if
statement that identifies the quantity of a small, medium, or large
package.
Performing this process from start to end then looks as illustrated below:
With just a few clicks you can create a column with your own logic.
Custom Column
In case you need more specific logic and feel comfortable writing code,
you can also create a custom column. The custom column allows you to
create custom formulas and functions. Imagine you have a table with
Countries and Product Prices Excluding VAT. For this table you want to
create a price that includes VAT, but each country has a different VAT
percentage.
Important
Unfortunately, you can’t create this if statement in a single step using the
Conditional Column dialogue box. But if you master creating conditional
if statements in Power Query this process is relatively easy.
To do that :
Combining Queries in Power Query can be useful when you have data
stored in multiple files, sources or queries that you want to combine into a
single table. Power Query provides two ways to combine data: Appending
Queries and Merging Queries.
Append Queries
In the below picture, the Sales March and Sales April data come from
different Excel files. They currently exist in different queries. Combine
them into a single query you can:
You are now left with a single table that contains sales data from both
March and April.
Merge Queries
For example, if you have sales data in one table and discount data in
another, you can use the Merge Queries operation to combine tables into
a single table.
To do that you:
Important
Both Appending and Merging Queries may require some data cleaning and
transformation to ensure that the data format and types are compatible
and match correctly. However, once you have mastered these functions,
combining data in Power Query becomes a breeze.
Advanced Editor
The Advanced Editor is a powerful tool in Power Query that allows you to
delve deeper into the M-code generated by your applied steps. The
Advanced Editor shows the M language code that Power Query generates
based on your transformations.
Tip
The Advanced Editor is useful for users who want more flexibility to
create transformations that can’t be done through the standard interface.
For beginners, it may seem intimidating, but it’s a useful tool to learn as
you become more familiar with Power Query Syntax.
To access the Advanced Editor you can click the Advanced Editor button
in the Home tab, as illustrated below:
This operation opens a menu that contains the M-code fo your query. An
example query could look like the image below:
So what do we see when looking at the advanced editor?
You will find that the Advanced Editor always opens with a let
statement. This indicates that the following lines contain
variables.
Once the variables (applied steps) have been defined, the word in
indicates the end of the sequence of variables.
The step that follows the in at the last line (AddLengthTest) is the
variable that Power Query will return.
Caution
To ensure that Power Query displays the Applied Steps pane, your query
must be formatted with the ‘let..in’ statement correctly. Begin the query
with ‘let’ and then list your variables. Conclude with the ‘in’ keyword,
followed by the final variable in the series. Referencing a different variable
at the end will cause the editor to show only the query result as a single
step, instead of the individual query steps.
Overall, the Advanced Editor is a valuable tool in Power Query that allows
you to create more complex and precise data transformations. Although it
may seem intimidating at first, it’s a tool that can help you gain a better
understanding of the Power Query M language and the full capabilities of
Power Query.
After you have completed the query and transformed the data in the way
that you want, you will need to save your work.
Close & Apply: Clicking close and apply loads your queries to the
desired destination. This can be an Excel Table, the Power BI data
model, a dataflow or any destination connected to your tool. This
operation also closes the Power Query Editor.
Apply: Performs the same action as Close & Apply, except it keeps
the power Query window open.
Conclusion
While there are some limitations to Power Query, including speed and
availability issues, it remains a valuable tool to help you transform your
data and generate insights that you can use for your business.
In this article, we’ve covered the benefits of Power Query, including its
easy learning curve, custom functions, and the ability to reshape data
easily. We’ve also looked at the Power Query components, including the
Power Query Editor, ribbon tabs, and how to connect to data sources.
Overall, Power Query is a free and accessible tool that can help anyone
work with data. And if you’re still not sure, here’s what I think about
learning Power Query, even if you know SQL.
Happy Querying!
Recommended Reading…
Advanced Topics
Share on:
Power Query
April 8, 2024
Power Query
April 4, 2024
Power Query
April 2, 2024
Power Query
April 2, 2024
10
Comments
Leave a Reply
1.
Matthew Boyd
Just wanted to give thanks for your guides. I had to learn Power
Query by Trial and Error. I’ve been trying to train others and your
guides do an excellent job of simplifying and explaining Power
Query.
Reply
2.
sweta sarkar
Reply
Rick de Groot
Reply
Julie
Reply
3.
Oscar
Reply
Rick de Groot
4.
It is really great job.Thank for your afford. Hope it will reach so many
people and help them.
Reply
Rick de Groot
Thanks Adem
Reply
5.
André Stolk
Reply
Rick de Groot
Reply
Leave a comment
Comment
Name Email
Save my name, email, and website in this browser for the next time I
comment.
This site uses Akismet to reduce spam. Learn how your comment data is
processed.
Take your Power Query skills to the next level with insider tips and
techniques.
Subscribe to my Channel
Master the M language?
Recommended Reading
Recent Posts
About BI Gorilla
BI Gorilla is a blog about DAX, Power Query and Power BI. It was founded
in 2018 by Rick de Groot with the goal to provide easy to understand
resources to help you advance.
Latest Posts
You can now find everything you need to know on the Power Query M
language (syntax, examples, articles etc.) at https://powerquery.how/.
Newsletter