Course 3
Course 3
Following are some data-collection considerations to keep in mind for your analysis:
Data sources
If you don’t collect the data using your own resources, you might get data from second-party or third-
party data providers. Second-party data is collected directly by another group and then sold. Third-
party data is sold by a provider that didn’t collect the data themselves. Third-party data might come
from a number of different sources.
Time frame
If you are collecting your own data, decide how long you will need to collect it, especially if you are
tracking trends over a long period of time. If you need an immediate answer, you might not have
time to collect new data. In this case, you would need to use historical data that already exists.
Use the flowchart below if data collection relies heavily on how much time you have:
Data formats in practice
When you think about the word "format," a lot of things might come to mind. Think of an
advertisement for your favorite store. You might find it in the form of a print ad, a billboard, or even a
commercial. The information is presented in the format that works best for you to take it in. The
format of a dataset is a lot like that, and choosing the right format will help you manage and use your
data in the best way possible.
Data format examples
As with most things, it is easier for definitions to click when we can pair them with real life examples.
Review each definition first and then use the examples to lock in your understanding of each data
format.
the
following table highlights the differences between primary and secondary data and examples of each
Data Format
Definition Examples
Classification
- Data from an interview you conducted - Data from a survey retu
Collected by a researcher
Primary data 20 participants - Data from questionnaires you got back from a gr
from first-hand sources
workers
- Data you bought from a local data analytics firm’s customer pro
Gathered by other people
Secondary data Demographic data collected by a university - Census data gathere
or from other research
federal government
the following table highlights the differences between internal and external data and examples of
each
Data Format
Definition Examples
Classification
- Wages of employees across different business units tracked by HR -
Data that lives inside a
Internal data Sales data by store location - Product inventory levels across
company’s own systems
distribution centers
Data that lives outside of a - National average wages for the various positions throughout your
External data
company or organization organization - Credit reports for customers of an auto dealership
the following table highlights the differences between continuous and discrete data and examples of
each
Data Format
Definition Examples
Classification
Data that is measured and can - Height of kids in third grade classes (52.5 inches, 65.7 in
Continuous data
have almost any numeric value Runtime markers in a video - Temperature
- Number of people who visit a hospital on a daily basis (1
Data that is counted and has a
Discrete data 200) - Room’s maximum capacity allowed - Tickets sold i
limited number of values
current month
the following table highlights the differences between qualitative and quantitative data and examples
of each
Data Format
Definition Examples
Classification
Subjective and explanatory measures - Exercise activity most enjoyed - Favorite brands o
Qualitative
of qualities and characteristics loyal customers - Fashion preferences of young adu
- Percentage of board certified doctors who are wom
Specific and objective measures of
Quantitative Population of elephants in Africa - Distance from E
numerical facts
Mars
the following table highlights the differences between nominal and ordinal data and examples of
each
Data Format
Definition Examples
Classification
A type of qualitative data that - First time customer, returning customer, regular customer -
Nominal isn’t categorized with a set applicant, existing applicant, internal applicant - New listing
order price listing, foreclosure
- Movie ratings (number of stars: 1 star, 2 stars, 3 stars) - Ra
A type of qualitative data with
Ordinal choice voting selections (1st, 2nd, 3rd) - Income level (low i
a set order or scale
middle income, high income)
Data Format
Definition Examples
Classification
Data organized in a certain format, like rows and - Expense reports - Tax returns - Store
Structured data
columns inventory
Data that isn’t organized in any easily identifiable
Unstructured data - Social media posts - Emails - Videos
manner
the following table highlights the differences between structured and unstructured data and
examples of each
Structured data
As we described earlier, structured data is organized in a certain format. This makes it easier to
store and query for business needs. If the data is exported, the structure goes along with the data.
Unstructured data
Unstructured data can’t be organized in any easily identifiable manner. And there is much more
unstructured than structured data in the world. Video and audio files, text files, social media content,
satellite imagery, presentations, PDF files, open-ended survey responses, and websites all qualify
as types of unstructured data.
Important note: As a junior data analyst, you won't be asked to design a data model. But you might
come across existing data models your organization already has in place.
Data-modeling techniques
There are a lot of approaches when it comes to developing data models, but two common methods
are the Entity Relationship Diagram (ERD) and the Unified Modeling Language (UML) diagram.
ERDs are a visual way to understand the relationship between entities in the data model. UML
diagrams are very detailed diagrams that describe the structure of a system by showing the system's
entities, attributes, operations, and their relationships. As a junior data analyst, you will need to
understand that there are different data modeling techniques, but in practice, you will probably be
using your organization’s existing technique.
You can read more about ERD, UML, and data dictionaries in this data modeling techniques article.
Data analysis and data modeling
Data modeling can help you explore the high-level details of your data and how it is related across
the organization’s information systems. Data modeling sometimes requires data analysis to
understand how the data is put together; that way, you know how to map the data. And finally, data
models make it easier for everyone in your organization to understand and collaborate with you on
your data. This is important for you and everyone on your team!
You will buy the shoes only if they are pink and grey
You will buy the shoes if they are entirely pink or entirely grey, or if they are pink and grey
You will buy the shoes if they are grey, but not if they have any pink
Below are Venn diagrams that illustrate these preferences. AND is the center of the Venn diagram,
where two conditions overlap. OR includes either condition. NOT includes only the part of the Venn
diagram that doesn't contain the exception.
The AND operator
Your condition is “If the color of the shoe has any combination of grey and pink, you will buy them.”
The Boolean statement would break down the logic of that statement to filter your results by both
colors. It would say “IF (Color=”Grey”) AND (Color=”Pink”) then buy them.” The AND operator lets
you stack multiple conditions.
Below is a simple truth table that outlines the Boolean logic at work in this statement. In the Color is
Grey column, there are two pairs of shoes that meet the color condition. And in the Color is Pink
column, there are two pairs that meet that condition. But in the If Grey AND Pink column, there is
only one pair of shoes that meets both conditions. So, according to the Boolean logic of the
statement, there is only one pair marked true. In other words, there is one pair of shoes that you can
buy.
Color is Grey Color is Pink If Grey AND Pink, then Buy Boolean Logic
Grey/True Pink/True True/Buy True AND True = True
Grey/True Black/False False/Don't buy True AND False = False
Red/False Pink/True False/Don't buy False AND True = False
Red/False Green/False False/Don't buy False AND False = False
The OR operator
The OR operator lets you move forward if either one of your two conditions is met. Your condition is
“If the shoes are grey or pink, you will buy them.” The Boolean statement would be “IF
(Color=”Grey”) OR (Color=”Pink”) then buy them.” Notice that any shoe that meets either the Color is
Grey or the Color is Pink condition is marked as true by the Boolean logic. According to the truth
table below, there are three pairs of shoes that you can buy.
Color is Grey Color is Pink If Grey OR Pink, then Buy Boolean Logic
Red/False Black/False False/Don't buy False OR False = False
Black/False Pink/True True/Buy False OR True = True
Grey/True Green/False True/Buy True OR False = True
Grey/True Pink/True True/Buy True OR True = True
The NOT operator
Finally, the NOT operator lets you filter by subtracting specific conditions from the results. Your
condition is "You will buy any grey shoe except for those with any traces of pink in them." Your
Boolean statement would be “IF (Color="Grey") AND (Color=NOT “Pink”) then buy them.” Now, all of
the grey shoes that aren't pink are marked true by the Boolean logic for the NOT Pink condition. The
pink shoes are marked false by the Boolean logic for the NOT Pink condition. Only one pair of shoes
is excluded in the truth table below.
Boolean Logic for NOT If Grey AND (NOT Pink), then
Color is Grey Color is Pink Boolean Logic
Pink Buy
Grey/True Red/False Not False = True True/Buy True AND True = True
Grey/True Black/False Not False = True True/Buy True AND True = True
Grey/True Green/False Not False = True True/Buy True AND True = True
Grey/True Pink/True Not True = False False/Don't buy True AND False = False
Whether you are doing a search for new shoes or applying this logic to your database queries,
Boolean logic lets you create multiple conditions to filter your results. And now that you know a little
more about how Boolean logic is used, you can start using it!
Transforming data
What is data transformation?
A woman presenting data, a hand holding a medal, two people chatting, a ship's wheel being
steered, two people high-fiving each other
In this reading, you will explore how data is transformed and the differences between wide and long
data. Data transformation is the process of changing the data’s format, structure, or values. As a
data analyst, there is a good chance you will need to transform data at some point to make it easier
for you to analyze it.
Long data is data where each row contains a single data point for a particular item. In the long data
example below, individual stock prices (data points) have been collected for Apple (AAPL), Amazon
(AMZN), and Google (GOOGL) (particular items) on the given dates.
With data transformed to wide data, you can create a chart comparing how each company's stock
changed over the same period of time.
You might notice that all the data included in the long format is also in the wide format. But wide data
is easier to read and understand. That is why data analysts typically transform long data to wide data
more often than they transform wide data to long data. The following table summarizes when each
format is preferred:
Data anonymization
What is data anonymization?
You have been learning about the importance of privacy in data analytics. Now, it is time to talk
about data anonymization and what types of data should be anonymized. Personally identifiable
information, or PII, is information that can be used by itself or with other data to track down a
person's identity.
Data anonymization is the process of protecting people's private or sensitive data by eliminating that
kind of information. Typically, data anonymization involves blanking, hashing, or masking personal
information, often by using fixed-length codes to represent data columns, or hiding data with altered
values.
Telephone numbers
Names
License plates and license numbers
Social security numbers
IP addresses
Medical records
Email addresses
Photographs
Account numbers
For some people, it just makes sense that this type of data should be anonymized. For others, we
have to be very specific about what needs to be anonymized. Imagine a world where we all had
access to each other’s addresses, account numbers, and other identifiable information. That would
invade a lot of people’s privacy and make the world less safe. Data anonymization is one of the
ways we can keep data private and secure!
Third-party data is collected by an entity that doesn’t have a direct relationship with the data. You
might remember learning about this type of data earlier. For example, third parties might collect
information about visitors to a certain website. Doing this lets these third parties create audience
profiles, which helps them better understand user behavior and target them with more effective
advertising.
Personal identifiable information (PII) is data that is reasonably likely to identify a person and make
information known about them. It is important to keep this data safe. PII can include a person’s
address, credit card information, social security number, medical records, and more.
Everyone wants to keep personal information about themselves private. Because third-party data is
readily available, it is important to balance the openness of data with the privacy of individuals.
Databases in data analytics
Databases enable analysts to manipulate, store, and process data. This helps them search through
data a lot more efficiently to get the best insights.
Relational databases
A relational database is a database that contains a series of tables that can be connected to show
relationships. Basically, they allow data analysts to organize and link data based on what the data
has in common.
In a non-relational table, you will find all of the possible variables you might be interested in
analyzing all grouped together. This can make it really hard to sort through. This is one reason
why relational databases are so common in data analysis: they simplify a lot of analysis
processes and make data easier to find and use across an entire database.
Database Normalization
Normalization is a process of organizing data in a relational database. For example, creating
tables and establishing relationships between those tables. It is applied to eliminate data
redundancy, increase data integrity, and reduce complexity in a database.
By contrast, a foreign key is a field within a table that is a primary key in another table. A table
can have only one primary key, but it can have multiple foreign keys. These keys are what create
the relationships between tables in a relational database, which helps organize and connect data
across multiple tables in the database.
Some tables don't require a primary key. For example, a revenue table can have multiple foreign
keys and not have a primary key. A primary key may also be constructed using multiple columns
of a table. This type of primary key is called a composite key. For example, if customer_id and
location_id are two columns of a composite key for a customer table, the values assigned to
those fields in any given row must be unique within the entire table.
Take a look at any data you find. What is it? Where did it come from? Is it useful? How do you
know? This is where metadata comes in to provide a deeper understanding of the data. To put it
simply, metadata is data about data. In database management, it provides information about other
data and helps data analysts interpret the contents of the data within a database.
Regardless of whether you are working with a large or small quantity of data, metadata is the mark
of a knowledgeable analytics team, helping to communicate about data across the business and
making it easier to reuse data. In essence, metadata tells the who, what, when, where, which, how,
and why of data.
Elements of metadata
Before looking at metadata examples, it is important to understand what type of information
metadata typically provides.
Examples of metadata
In today’s digital world, metadata is everywhere, and it is becoming a more common practice to
provide metadata on a lot of media and information you interact with. Here are some real-world
examples of where to find metadata:
Photos
Whenever a photo is captured with a camera, metadata such as camera filename, date, time, and
geolocation are gathered and saved with it.
Emails
When an email is sent or received, there is lots of visible metadata such as subject line, the sender,
the recipient and date and time sent. There is also hidden metadata that includes server names, IP
addresses, HTML format, and software details.
Websites
Every web page has a number of standard metadata fields, such as tags and categories, site
creator’s name, web page title and description, time of creation and any iconography.
Digital files
Usually, if you right click on any computer file, you will see its metadata. This could consist of file
name, file size, date of creation and modification, and type of file.
Books
Metadata is not only digital. Every book has a number of standard metadata on the covers and
inside that will inform you of its title, author’s name, a table of contents, publisher information,
copyright description, index, and a brief description of the book’s contents.
Organization guidelines
This reading summarizes best practices for file naming, organization, and storage.
Work out and agree on file naming conventions early on in a project to avoid renaming files
again and again.
Align your file naming with your team's or company's existing file-naming conventions.
Ensure that your file names are meaningful; consider including information like project name
and anything else that will help you quickly identify (and use) the file for the right purpose.
Include the date and version number in file names; common formats are YYYYMMDD for dates
and v## for versions (or revisions).
Create a text file as a sample file with content that describes (breaks down) the file naming
convention and a file name that applies it.
Avoid spaces and special characters in file names. Instead, use dashes, underscores, or capital
letters. Spaces and special characters can cause errors in some applications.
Create folders and subfolders in a logical hierarchy so related files are stored together.
Separate ongoing from completed work so your current project files are easier to find. Archive
older files in a separate folder, or in an external storage location.
If your files aren't automatically backed up, manually back them up often to avoid losing
important work.
In order to do this, companies need to find ways to balance their data security measures with their
data access needs.
Luckily, there are a few security measures that can help companies do just that. The two we will talk
about here are encryption and tokenization.
Encryption uses a unique algorithm to alter data and make it unusable by users and applications that
don’t know the algorithm. This algorithm is saved as a “key” which can be used to reverse the
encryption; so if you have the key, you can still use the data in its original form.
Tokenization replaces the data elements you want to protect with randomly generated data referred
to as a “token.” The original data is stored in a separate location and mapped to the tokens. To
access the complete original data, the user or application needs to have permission to use the
tokenized data and the token mapping. This means that even if the tokenized data is hacked, the
original data is still safe and secure in a separate location.
Encryption and tokenization are just some of the data security options out there. There are a lot of
others, like using authentication devices for AI technology.
As a junior data analyst, you probably won’t be responsible for building out these systems. A lot of
companies have entire teams dedicated to data security or hire third party companies that specialize
in data security to create these systems. But it is important to know that all companies have a
responsibility to keep their data secure, and to understand some of the potential systems your future
employer might use.