Analyze and Make Data Driven Recommendations

Interpret Statistical Output

In this course, we discuss ways to analyze and interpret the output of a regression analysis.

Use Regression Analysis

This lesson prepares you to review the outputs of a regression analysis and explore how
regression models can help guide business decisions.

Regression analysis
Regression analysis is a tool that helps you to make inferences and predictions about data. It’s
an observational method that can help you to identify correlative results in the data. It also
contains metrics that help you to understand the accuracy of its predictions.

You can run regression analysis in R, Python and many other programming languages and
data analytics tools.

There are several types of regression analyses. The type of analysis you choose depends upon
the business context. For example, logistic regression can be helpful when the dependent
variable has a predetermined set of possible values or can only be a true or false value. Linear
regression may be more helpful to analyze data where there are no restrictions on the value of
the dependent variable, for example, return on ad spend. In this lesson, we focus on simple
linear regression and multiple linear regression.

Axion scenario
Moh is a data analyst at Axion, a fictitious manufacturer of premium televisions that
feature the patented Axion high-definition screen technology.

The marketing team ran a multi-channel ad campaign, which included ads on Facebook
platforms that promoted the latest Axion television model. They then asked Moh to
analyze the results of that campaign.

Simple linear regression and multiple linear

regression
Linear regression analysis helps to assess the strength of the relationship between a dependent
variable and an independent variable. It can also help determine whether the relationship
between a set of independent variables follows a pattern.
An independent variable is a value that changes as a result of independent factors outside of
your data set.

A dependent variable is a value that changes because of a hypothesized change in an

independent variable. With regression analysis, you can test this hypothesis.
Simple linear regression analyzes the relationship between one dependent variable and one
independent variable.

As an example scenario, Moh uses simple linear regression to analyze the relationship
between the number of ad impressions and the value of Axion sales in a given period. The
number of ad impressions is the independent variable and the values of sales is the dependent
variable.

Dependent variable
Independent variable
Multiple linear regression analyzes the relationship between one dependent variable and
multiple independent variables.

As an example scenario, the marketing team at Axion has a hypothesis that people in North
America are more likely to buy electronics on days when the weather is cold. To analyze the
relationship between ad impressions, the outdoor air temperature in a given city and the value
of sales in a day, Moh uses multiple linear regression.

Dependent Variable
Independent Variable A
Independent Variable B

Scatterplots
A scatterplot is an exploratory data visualization chart. It helps you examine data to identify
patterns or show where gaps or outliers might exist in your data. It's often a starting point for
data analysis, specifically regression analysis.

You can use a scatterplot to view the results of a simple linear regression analysis. The
independent variable is on the x-axis, and the dependent variable is on the y-axis. The
regression line in the middle represents the predicted values of Y given the changes in X.

Impressions – Dependent variable (Y)

Ad spend - Independent variable (X)
Correlation coefficient
The correlation coefficient is the rate of change in the dependent variable per unit change
in the independent variable. The correlation coefficient is often known as r. r always has a
value between -1 and 1. An r value close to 1 means that there's a strong positive
relationship between the two variables. Conversely, an r value closer to -1 means that
there's a strong negative relationship between the two variables.

You don't need to learn how to calculate r. Your regression analysis software does this for
you. Let's look at some simple examples.

 In first scatterplot chart, the data points are in perfect correlation to the regression
line. The value of r in this analysis is 1.
 In second chart, the data points are scattered in the direction of the line, but there's
a wide variation between some points and the line. The value of r in this analysis is
0.3.
 In third chart, the data points are scattred across the entire area of the chart. There
is no correlation between these data points. The value of r in this analysis is 0. This
data isn't suitable for use in a regression analysis.

Positive coefficients and negative coefficients

The sign of each coefficient indicates the direction of the relationship between the
independent variable and the dependent variable.

A positive coefficient for an independent variable means that an increase in the independent
variable correlates to an increase in the dependent variable. A positive coefficient creates an
upward diagonal line on a scatterplot.

Similarly, a negative coefficient means that an increase in the independent variable correlates
to a decrease in the dependent variable. This negative relationship creates a downward
diagonal line on a scatterplot.

View data correlation with scatterplots

Data on a scatterplot provides a visual representation of the correlation between two
variables. The position of the data points and the direction of the regression line indicate the
value of the correlation coefficient.

In a perfect correlation, all data points are situated on the line. In a high correlation, all
data points are close to the line, and in a low correlation, the points are further away from
the line. A positive correlation line slopes upwards, while a negative correlation line
slopes downwards.

Analyze the effect of a single independent variable

Let's look at a simple linear regression analysis that Moh has prepared.

The purpose of this analysis is to examine the correlation between ad impressions and
product sales. The marketing team would like to predict future sales based on the number of
ad impressions.

Moh uses a scatterplot to present the results of his simple linear regression analysis.

The scatterplot shows ad impressions on the x-axis and sales on the y-axis.

In the scatterplot, Moh can see a positive relationship between ad impressions and sales. The
scatterplot shows that in this data set, an increase in ad impressions correlates with an
increase in sales.

Regression analysis outputs

People typically perform regression analyses with analysis software, such as R or Python,
that read data directly from a database. You can also perform a regression analysis in Excel,
but there are limits on the number of data rows that Excel can process.

A typical regression analysis outputs a set of values, which are similar to those in the image.
You don't need to know what all of these values mean. We'll discuss the most important
values in the analysis output.

The values in the output provide information about the relationship between the variables and
the accuracy of the predictions in the regression model. The constant variable is the rate of
the change in the dependent variable when there's no change in either of the independent
variables.

Some of the output values are useful to help you understand the statistical relationship
between variables. Others are useful when you compare models between one another to
measure how well the models capture the true relationships between these variables.
Axion example
As the marketing team at Axion creates their media budget for their new campaign, they want
to estimate how increases in media spending will affect product sales.

Meanwhile, the product management team wants to estimate the effects of a proposed
increase in product prices. The business could make more money with higher prices, but the
team knows there's a risk that increased prices may lead to decreased sales.

Both teams ask Moh for data analyses to help guide their decisions, so he decides to use a
multiple linear regression model.

This table shows the regression analysis output. This is fictitious data, so some of the values
you see ould be much higher or lower than you'd see in the real world.

The constant, which is the rate of increase in sales without a change in media spend or prices,
is 0.000565. spend or prices, is 0.000565.
 The media spend coefficient is positive, which means the model predicts that for each
additional unit of media spend, product sales should increase by 1.031918. In the model, this
means that for every additional $100 the Axion team spends on media, they can expect to sell
approximately 1.031918 more units of the new television.

The price coefficient is negative, which means the model predicts that for each unit of
increase in prices, sales of the television should decrease by 0.483421 units.

Moh runs a multiple linear regression analysis to identify if there's a relationship between
Facebook ad spend, the weather and sales of Axion televisions.

Product sales is the dependent variable, and Facebook ad spend and the outdoor air
temperature in the city are the independent variables. Facebook ad spend is measured in
units of $100. The coefficient for Facebook ad spend is 4.567. What can Moh predict
from this model?

For every $100 increase in Facebook ad spend, Axion can expect to sell 4.567 fewer
televisions.

When the price of Facebook ads increases by $4.567, Axion can expect to sell one
extra television.

For every $100 increase in Facebook ad spend, Axion can expect to sell an extra 4.567
televisions.

When the weather is 4.567 degrees warmer, Axion can expect to sell an extra 4.567
televisions.
Facebook ad spend is an independent variable in this model. The coefficient of an independent variable
predicts the change in the dependent variable for each unit change in the independent variable. This means that
Axion can expect to sell 4.567 more televisions for each $100 increase in ad spend.

Key takeaways
 Regression analysis can help you to determine the strength of the relationship between a
dependent variable and one or more independent variables.
 The correlation coefficient is the rate of change in the dependent variable per unit change in
the independent variable.
 A scatterplot can help you visualize the results of a simple linear regression analysis.

Sources
 Khan Academy - Statistics and probability
Linear Relationships in Data
This lesson prepares you to explore the concepts of best fit lines and least squares regression,
including how regression lines on scatterplot graphs are calculated.

Find the best fit line

Regression analysis is only effective when there's a linear relationship between the value of
the independent variable and the value of the dependent variable. Some outlier values are
acceptable, but they might reduce the accuracy of the model.

The best fit line minimizes the distance between data points and the line. Let's examine two
examples of best fit lines.

In this chart, there's a linear relationship

between the data points. This data is suitable
for regression analysis.

In this chart, there are clusters of values in

different locations. The regression line
ignores some of the values. You can't ignore
values in your data set though, because the
results of your analysis would be inaccurate.

Your regression analysis software can help you to automatically find the best fit line. You can
look at the positions of the data points on the chart to determine if there is a linear
relationship between the values.

Least-squared correlation
The least squares method, also known as the residual error method, is the most common way
to identify the best regression line.

When you run a least squares regression model, the model calculates two values (the
intercept value and the slope value) that minimize residual errors. The lower the
residual error, the better the model. You don't have to manually calculate these values.

Your regression analysis software performs the calculations for you.

Slope and intercept

Here's a scatterplot with seven data points and a regression line. The regression line shows
the predicted number of conversions in relation to ad spend value, which is the independent
variable. The number of conversions is the dependent variable, which the model predicts to
increase when ad spend increases.

Your regression analysis software populates values into an equation when it calculates the
regression line. You can use the regression line equation to predict values along the line,
including values beyond the range shown on the chart.

The regression line equation is Y = a + bX.

Y is the number of conversions, as the y-axis of the chart shows. X is the value of ad spend, as
the x-axis of the chart shows.

a is the intercept value, which is the value of Y when X is 0. The intercept on this chart is 55.
In business terms, this could mean that 55 conversions occur each day when ad spend is $0.

b is the slope value, which is the change in Y when you change one unit of X. In the
equation, bX means b multiplied by X. The slope on this chart is the increase in conversions
per dollar of ad spend. In this simple example, we can interpret the slope value as the
coefficient for X.
 Slope, intercept and regression lines
The regression line always starts at the intercept point on the y-axis of a chart. When the
slope value changes, the angle of the line changes.

In each example, the intercept value (a) and the increase in ad impressions (X) are the same.
The slope value (b) changes. The increase in b causes an increase in the angle of the
regression line.

***
Moh runs a regression analysis on an Axion Smart TVs ad campaign to find the
relationship between Facebook ad spend and product sales.

Moh uses the regression line equation Y = a + bX to predict future sales. Y is the increase
in product sales.

Moh inputs these values into the equation:

 a (intercept) = 100

 b (slope) = 3

 X (increase in ad spend)= $100

How many additional sales does the model predict?

400
100
-200
300
 Key takeaways
 The best fit regression line minimizes the distance between data points and the line.
 The least squares method, also known as the residual error method, is the most common way
to identify the best regression line.
 Your regression analysis software uses slope and intercept to calculate the best fit regression
line. Intercept is the value of the dependent variable before any change in the independent
variable. Slope is the change in the dependent variable per unit change in the independent
variable.

Sources
 Khan Academy - Statistics and probability

Determine Model Accuracy

This lesson prepares you to use R-squared and adjusted R-squared values to determine the
accuracy of your regression model.

Introduction to R-squared
R-squared (which can also be written as R2) is the value that tells you how accurate your
model is. Simple linear regression models use R-squared and multiple linear regression
models use adjusted R-squared.

R-squared tells you how much of the change in the dependent variable was related to the
change in the independent variable. Most regression models don't show an exact correlation
between the independent variable and the dependent variable. There are always residual
errors, which are changes related to influences outside of your model.

Adjusted R-squared considers the correlation between multiple independent variables and the
value of the dependent variable. Just like R-squared, it tells you how much of the change in
the dependent variable was related to the change in the independent variables.
You don't need to learn the formula for adjusted R-squared right now. The adjustment is a
calculation your regression analysis software can perform for you.

Use R-squared to determine model accuracy

Let's return to our example scenario of Moh, who works as a data analyst at a fictitious
electronics manufacturer called Axion.

Moh has run a multiple linear regression model to analyze the effect media spend and price
increases have on sales of the new Axion television. He needs to provide evidence to the
product and marketing teams that the model provides accurate predictions based on their
business data. An accurate model is considered to be a good fit for the data.
This is a multiple linear model, meaning a model with multiple independent variables, so
Moh looks at the adjusted R-squared value. Adjusted R-squared for this model is 0.998515.
This means that the changes in the independent variables can explain approximately 99% of
the change in the dependent variable. This R-squared value indicates that the model is a good
fit for the business data.

Compare R-squared values between models

Moh runs a set of simple linear regression analyses to predict the effect that individual
variables have on the sale of Axion televisions.

The models have these R-squared values:

 0.981
 0.9024
 0.856
 0.32

Based on these R-squared values, which models are a good fit for the data?

Drag the R-squared values onto the correct answers.

Good fit
R-squared = 0.981 R-squared = 0.9024 R-squared = 0.856

Bad fit
R-squared = 0.32

A low R-squared value means that the model is probably not a good fit for the business data.
Predictions in this model may not be reliable.Kn
owledge check
Moh uses a simple linear regression analysis to identify the relationship between sales of
the new Axion television and the previous model in the range.

The independent variable is the number of the new television model sold each day and the
dependent variable is the number of old television model sold each day. Moh reviews the
analysis output. Which value in the analysis output indicates the accuracy of the
predictions?
Adjusted R-squared
 Durbin-Watson statistic
R-squared
Schwartz criterion

Key takeaways
 R-squared is the measure of how accurate your model is. Simple linear regression models
use R-squared. Multiple linear regression models use adjusted R-squared.
 R-squared tells you how much of the change in the dependent variable was related to the
change in the independent variables.
 Adjusted R-squared considers the correlation between multiple independent variables and
the value of the dependent variable.
Sources
 Khan Academy - Statistics and probability

Measure the Success of a Regression Analysis

This lesson prepares you to analyze and interpret the statistical output of a regression
analysis.

Introduction to standard error

The standard error estimates the standard deviation of the coefficient. Standard deviation
is the measure of how close the values in a data set are to the mean value. The coefficient
is the predicted change in the value of the dependent variable per unit change in the
independent variable.

You can use the standard error to understand how precisely the model estimates the value
of the coefficient. The smaller the standard error is, the more precise the estimate for that
coefficient is as well.

Standard error and the precision of a model

Let's return to our example scenario of Moh, a data analyst at Axion, a fictitious
electronics manufacturer.

Moh has run a multiple linear regression model to analyze the effect of media spend and
price increases on sales of the new Axion television. His hypothesis states that media
spend has a statistically significant influence on sales.
This table shows the outputs the regression analysis software Moh uses has generated.
This is fictitious data, so some of the values you see here could be much higher or lower
than you'd see in the real world.

The output shows the standard error for each coefficient. The media spend variable has a
low standard error, which means the model is a relatively precise measure for the change
in sales per unit of media spend.

Introduction to t-stat and p-value

The t-stat is the coefficient divided by the standard error. The model uses the t-stat to
calculate the p-value, which is sometimes referred to in a model as the probability or 'prob.'

The p-value is important, because it tells you whether the coefficient is statistically
significant. Statistical significance helps you to understand the strength of the relationship
between the independent variable and the dependent variable. You can interpret the p-value
as the confidence level.

For example, to have a confidence of 95%, we use a p-value of (1 - 0.95) = 0.05.

When the p-value is lower than 0.05, there's generally sufficient evidence that the coefficient
prediction is statistically significant, and you can consider the independent variable to be a
reliable predictor for the dependent variable.

If the p-value is greater than 0.05, it could indicate that the independent variable isn't a
reliable predictor for the dependent variable.

p-value and statistical significance

In our example scenario, Moh looks at the p-values in the regression analysis output to
determine if the model provides evidence that proves his hypothesis. The hypothesis
states that media spend is a statistically significant predictor of sales.
In this model, the independent variables, media spend and prices each have a p-value
lower than 0.05. This means that both independent variables are statistically significant.
Moh now knows that the media spend coefficient is a reliable predictor of sales when
there's a change in media spend. He also now knows that the same is true for product
prices.

The constant p-value is greater than 0.05, which means that it’s not statistically
significant. A high constant p-value could occur, for example, with a new product that
doesn’t yet have established baseline sales. The new television model is a recent addition
to the Axion product range, so there's limited historical sales data available.

Compare t-stat and p-value between models

Moh runs a multiple linear regression model to compare the effects of different variables
on the sales of another Axion product, smart televisions. Here are the t-stats and p-values
from those analyses. Which variables are statistically significant in these models?

Variable: Ad impressions Variable: Video ad views Variable: Media spend

t-stat: -57.869 t-stat: 1.6735 t-stat: 177.356
p-value: 0.0296 p-value: 0.5603 p-value: 0.0001

Statistically significant Not statistically significant Statistically significant

Correct. A p-value lower than 0.05 is usually considered to be statistically significant. In this model,
the ad impressions and media spend variables are statistically significant.

Here's a review of the regression analysis that shows the effect of media spend and price
increases on sales of Axion televisions. Use this analysis output to answer the questions.
 As a data analyst, which interpretations can you make from this statistical output?
The constant variable isn't statistically significant in the model.
The prices variable is statistically significant in the model.
The prices variable isn't statistically significant in the model.
The media spend variable isn't statistically significant in the model.
Which variable in the regression analysis has the most precise coefficient?
Media spend All coefficient have the same level precision
Constant Prices
Key takeaways
 Standard error helps you to determine how precisely the model estimates the value of the
coefficient.
 p-value tells you whether the coefficient is statistically significant.
 A p-value is higher than 0.05 could indicate that the independent variable isn't a reliable
predictor for the dependent variable.

Generate Insights to Make Data-Driven

Recommendations
In this course, we'll learn how to extract insights from data and how to use insights to make
data-driven recommendations.

Analyze Campaign Performance Data

This lesson prepares you to analyze campaign performance data and market research data
sources.

Generate actionable insights from data

Data  Insights  Recommendations
As a marketing data analyst, it’s often your responsibility to translate data into actionable
insights and make recommendations to your stakeholders.

Your initial insights can come from campaign performance and the results of tests or
experiments, such as Brand Lift, Conversion Lift and A/B tests. You can also supplement
these results with business data from sales order systems or CRM systems. Insights and
research reports about the target audience or brand vertical can also guide campaign
recommendations. Facebook IQ, for example, is a good resource for relevant insights from
market research. When you combine information from multiple sources to gain insight,
you’re in a stronger position to deliver a data-driven recommendation.

Analyze Facebook campaign performance

Ads Manager and Business Suite (formerly Business Manager) can help you evaluate the
results of your Facebook campaigns. Regardless of which tool you use to manage your ad
account, you can access data about the campaigns that you run on Facebook, including how
much you spent and how many people saw and clicked on your ads.

When you allow Facebook to use data related to actions that people took on your website, we
can provide conversion insights for your app or store. Ad impressions and ad clicks are
related to your conversion data when you use the Facebook pixel on your website, the
Facebook SDK in your app or the Conversions API. Conversions insights are attribution
results, which means that they're correlative, not causative.

Review campaign performance alongside business data

Business systems, such as sales order systems and CRM systems, can provide valuable
source data for marketing insights. This data can help generate insights about sales cycles,
best performing products and customer preferences. You can also learn about customer
demographics at an aggregated level and use that information to guide audience selections for
future campaigns.

Get insights with Facebook IQ

Facebook IQ can help you understand consumer behavior, create effective marketing
strategies and transform how your business reaches people. Its original research reports and
interactive tools can give you insights to guide your recommendations.

Facebook IQ tools and resources combine actionable insights from our internal data analyses
with commissioned market research from external research partners.

With Facebook IQ, you get:

 Insights articles: A library of long-form articles, videos, infographics and reports that can
give you new perspectives about audiences, industries and marketing trends.

 Interactive reports and white papers: Research studies and downloadable reports that
cover large data sets with customizable views.
  Consumer insights tools: Including Insights to Go statistics, Hot Topics trend rankings
and holiday season insights.

Enrich your analysis with third-party insights

In addition to Facebook IQ resources, you can also use census data, as well as third-party
reports as part of your research, including reports from the government, global market
research and advisory firms and market analysts in specific industries.

Storx scenario
Lana is a data analyst on the marketing team at Storx, a fictitious online baby and toddler care
emporium. Storx sells high-quality products, including organic formula, eco-friendly diapers,
fragrance-free wipes, teething rings, toys and more.

The brand runs regular ad campaigns on Facebook platforms. Lana is responsible for
analyzing campaign performance and making recommendations for future campaigns.
 Lana uses Business Suite to analyze the results of a recent campaign, which showed ads to
people who are interested in content related to babies and parenting. She sees a trend in the
age range of customers who react to the Storx ads, which suggests that a younger
demographic of new parents are interested in Storx products.
 Lana runs a report on the Storx customer relationship management system to get aggregated
data about customers who made their first Storx purchase during the campaign period. The
report validates her insight that Storx products have increased in popularity with new parents
ages 18-30.
 Lana wants to gather more evidence to support her campaign strategy recommendation. A
Facebook IQ research report provides statistics about how new parents search for baby-
related information on social media. She selects some insights from the study to include in
her stakeholder presentation.
Gather insights from multiple sources
Lana needs data to answer some marketing business questions. Which types of
information sources should she use?

Internal business data

Last holiday season, which Storx products sold best?
Facebook IQ
How many people viewed Storx new video ad?
Third-party research
How do new parents discover information about baby products?

Lana has the following information about Storx quarterly sales results.

Previous year (with Facebook campaigns during Q1-Q2)

Q1 Q2 Q3 Q4
 Previous year (with Facebook campaigns during Q1-Q2)

1,454,000 1,500,000 1,300,000 2,000,000

Actual year (with Facebook campaigns beginning in Q2)

Q1 Q2 Q3 Q4

1,400,000 1,600,000 1,350,000 1,350,000

What can Lana conclude about the observable behavior for year-over-year brand sales?

Sales increased 6.7% during Q2, which could be the result of Facebook campaigns.
Sales decreased during Q1 due to the lack of Facebook campaigns.
Sales during Q3 would have increased more if Facebook campaigns had been active
earlier.
Sales increased 10% during Q4 due to Facebook campaigns.
Key takeaways
 Ads Manager and Business Suite can help you evaluate the results of your campaigns on
Facebook.
 Facebook IQ can help you learn about trends and transform how your business reaches
people.
 Review campaign performance alongside business data.

Sources
 Ads Manager
 Business Suite
 Facebook IQ
Generate Insights With Data Analysis
This lesson prepares you to gather insights from the results of tests and experiments.

Data from tests and experiments

To assess Facebook campaign performance, you can run experiments or tests, such as Brand
Lift, Conversion Lift and A/B tests.
Brand Lift and Conversion Lift studies help you determine the incremental lift of your
campaign. These are randomized control trials, so their results are causative. A/B tests enable
you to test different versions of your ads against each other to help you optimize future
campaigns. The results of these tests are correlative.

This lesson focuses on the insights that you can gain from these tests and experiments.

Confidence levels and confidence intervals

The confidence level, expressed as a percentage, indicates how confident you can be that
your test or experiment will return the same results if you run it again. A confidence level of
95% or higher usually indicates a reliable set of results.

The confidence interval is the range of values that you can expect the estimate to fall between
if you repeat the test or experiment again, based on the confidence level.
Here's an example of the results from a Conversion Lift test:

This experiment found that conversions increased 5.5% among people who saw the ad
campaign. There's a 95% chance that if you repeated the experiment, the Conversion Lift
would be in the 2-7.5% range.

When a confidence interval contains a negative value, it could be evidence of a statistically

insignificant result. For example, a confidence interval of (-0.020, 0.050) could demonstrate a
statistically insignificant result.

You don't need to know how to calculate these values, since your statistical analysis software
does the math for you.
Insights from a Brand Lift test
The brand marketing team at Storx, a fictitious baby supplies business, runs a Facebook
brand awareness campaign to promote their free TeleNanny service, which lets new parents
chat with childcare experts through the Storx app or website.

To test the effectiveness of the campaign, the team runs a single-cell Brand Lift test and gets
150 responses for each test and control group. The test results are:
Lana, the data analyst for Storx, reviews the results and reports her insights to the marketing
team. She notes that the results are statistically significant at the 80% confidence level, which
is less than the target 95% confidence level. Brand awareness is slightly higher among the
test group than the control group. If the team wants to increase their confidence in the test,
they need to increase the target sample size. A larger sample increases the statistical power of
the test and its ability to detect lift.
Insights from a Conversion Lift test
The team at Storx ran three Conversion Lift tests last quarter. These tests measured
conversion rates for a test group of people who saw Storx ads on Facebook platforms and a
control group of people who didn’t see the ads.

Test 1 Test 2 Test 3

Test group conversion rate Test group conversion rate Test group conversion rate
35% 30% 35%
Control group conversion rate Control group conversion rate Control group conversion rate
20% 20% 20%
Confidence level Confidence level Confidence level
65% 95% 60%

Lana reviews the results and concludes that test 2 is most reliable because it shows a 10%
difference in lift between the test and control groups. It also has a higher confidence level
than test 1 and test 3. In this instance, the confidence level is the most important result. While
tests 1 and 3 show higher lift between the test and control groups, their confidence levels
indicate that these results aren’t reliable. Lana advises the marketing team to run test 1 and
test 3 again if they need a reliable comparison between the three.
Insights from an A/B test
An A/B test compares the performance of two ads or ad sets against each other. The winner is
the ad or ad set that has the lowest cost per result.

To determine the winner of an A/B test, we compare the cost per lead of each ad or ad set
based on the campaign objective. We also report a confidence level to indicate the likelihood
that the same results would occur if you ran the test again.
The Storx team runs an A/B test to test the hypothesis that a campaign to increase leads
performs better with video than with a static image.

Lana reviews the results and determines a strong result in favor of video. The test proves the
hypothesis with a 95% chance that the same campaign would win in a repeat test.

Multi-cell Conversion Lift experiments

The Storx marketing team runs a multi-cell Conversion Lift experiment to determine whether
a campaign generates sales lift.

Lana examines the results to discover which cell performed best. She explains the test results
and the predicted outcomes of future tests to her team.

Cell A: Sales lift = 4.0% 90% confidence interval = (0.010, 0.070)

The results show positive lift.
The confidence level is below the preferred level.
The confidence interval shows that the result of a repeat test would be positive lift.

Cell B: Sales lift = 4.5% 95% confidence interval = (-0.010, 0.084)

The results show positive lift.
The confidence level is good.
The confidence interval shows that the result of a repeat test could be negative lift or positive
lift.
Both cells show positive lift. Cell A has a confidence level of 90%, which is below the preferred
level of 95%. The cell B confidence interval isn’t always greater than 0, which shows a possibility of
negative lift upon a repeat test.

Storx scenario
Campaign performance data
Lana wants to learn how reach affects Storx incremental return on ad spend (ROAS) results.
She gathers historical campaign performance data and creates a scatterplot to compare reach
and ROAS for each campaign over the past 12 months. The marketing strategy changed
frequently over the past year, with adjustments to targeting, bidding strategies and
optimization metrics.

This scatterplot shows the test results. They're correlative, not causal, but from this chart,
Lana infers that an increase in ad reach may lead to increased ROAS. She decides to run a lift
test on the next campaign to gather more data.

Business systems data

Lana also uses data from internal business systems to gather insights. Storx ran a series of
Facebook ad campaigns to offer subscription boxes at a discount for the first three months,
which successfully attracted new customers. However, customers tended to cancel their
subscription after the discount period. As a result, the ROAS was lower than anticipated.

From the sales data, Lana infers that the discount promotion may have led to an issue with
brand value perception.

Multi-channel attribution
Marketing mix models (MMM)
Advertisers use marketing mix models (MMM) to attribute incremental sales to marketing
and non-marketing activities. A typical MMM is an analysis of historical activity, with
weekly observations of each marketing channel through a single variable. An advertiser
would use an MMM when they need to measure online and offline marketing channels
together, for example, to measure Facebook alongside television and print ads.
 Multi-touch attribution (MTA)
Advertisers use MTA to attribute credits for conversions to different digital channels along
the path to purchase. These models can take various forms, such as Last Click, Even Credit or
Time Decay. An advertiser would use MTA when they’re focused only on digital channels.

***
The Storx team runs monthly individual Conversion Lift tests across digital channels. The
latest tests show that channels A, B and C had statistically significant positive lift results
whereas channel D didn’t have statistically significant results.

The current campaigns target ads to existing Storx customers. The team plans to run future
campaigns across the same channels to attract new customers. To gain efficiencies in the
campaign budget, they want to find out how much overlap exists between the digital
channels. Which measurement approach most effectively identifies overlap between ad
channels?

Use a marketing mix model (MMM) to Identify correlations in daily sales data between
channels.
Use last-touch attribution to avoid double-counting conversions captured on multiple
platforms.
Use multi-touch attribution (MTA) to identify where multiple platforms contribute to
conversions.
Run future tests in which audiences don’t overlap, so that there’s no ability to see ads on
multiple platforms.
Multi-touch attribution (MTA) measures the incremental sales that each touchpoint generates during the path
to conversion rather than assigning full credit to a single touchpoint. This approach to measurement can
demonstrate the overlap between ad exposure on different channels.

Key takeaways
 Assess Facebook campaign performance with experiments or tests, such as Brand Lift,
Conversion Lift and A/B tests.
 Confidence levels demonstrate the chances that a repeat test would elicit the same result.
 Multi-touch attribution (MTA) helps identify overlap between ad channels.

Sources
 Khan Academy: Interpreting confidence levels and confidence intervals
 Measuring Facebook accurately in marketing mix models (PDF)

Make Data-Driven Recommendations

This lesson prepares you to make ad campaign recommendations based on data and insights.

Make recommendations to meet stakeholder needs

Your insights, gathered from campaign results, business data and relevant research, enable
you to develop actionable recommendations for future campaigns. You may need to gain
approval from business stakeholders, in particular those who manage budgets, before you
implement your recommendations.

When you present your recommendations, speak in the same terms as your stakeholders
and think about how to tell a compelling story.

Keep the needs of your stakeholders in mind. Consider what you want them to know, feel
and do. This helps you communicate in ways that they understand. Use only the most
relevant facts from your gathered insights.

What your stakeholders should know, feel and do

Know
To start your presentation with stakeholders, first restate your hypothesis, then present
evidence that proves it.
Introduce stakeholders to the subject matter with audience and market insights from
Facebook IQ and other sources.
Consider evidence you might include that's specific to the brand, for example, results
from tests and experiments, such as Brand Lift studies.

Feel
Think about your individual stakeholders and how you want them to feel.
Perhaps you want to evoke a sense of pride when they see your vision for the future of the
brand, or excite them about potential growth.
Consider parameters that are important to them, such as time, budget and creative. You
might also consider the prior experience stakeholders have with Facebook ads or other
forms of digital marketing.

Do
Your insights and recommendations should lead toward a request to your stakeholders to
take action. These actions should be within their control and should connect to their
business goals. Be clear about what you want them to do and the effects you expect those
actions to have.
Consider the actions you want them to take immediately after your presentation as well.
Communicate clear next steps to execute your plan and tell stakeholders about the short-
and long-term outcomes that you anticipate.

Storx scenario
Let's look at an example scenario. The team at Storx, a fictitious online baby supply store, is
preparing to launch a new line of baby clothing. Lana, a marketing data analyst, writes a
presentation to get approval for the marketing budget.
She focuses her presentation on what she wants stakeholders to know, how she wants them to
feel and what she wants them to do.
 First, Lana uses results from recent Brand Lift studies to show that Storx is recognized as a
quality brand for discretionary spend products for children.
She also references Conversion Lift studies for other product lines that show a correlation
between Facebook ad views and intent to purchase.
Next, Lana finds Facebook IQ insights that show trends about how parents discover new
brands on social media.
She wants to help stakeholders feel confident in their decision to run ad campaigns on
Facebook platforms.
Lana concludes her presentation with a clear recommendation, including the campaign
objective, duration, target audience, creative strategy and campaign budget.
She includes an estimated ROAS based on previous campaigns and asks stakeholders to
approve the new campaign.

Know, feel, do
The Storx marketing team hires an agency to produce video ads. Their marketing objective is
brand reach. Based on outcomes of previous campaigns and Brand Lift tests, the team has
these insights:

 Mix of short videos and static formats: 0.5-point lift, 14 million people reached and an
average cost of $30,000
 Long video formats: 2-point lift, 3 million people reached and an average cost of $50,000

The team wants to ask stakeholders to approve a new campaign. Consider what stakeholders
need to know, how they should feel and what the team wants them to do.

Know A Facebook IQ research report shows how video can increase ad recall, brand
awareness and purchase consideration.
Do We propose a new brand reach campaign with a mix of short videos and static images. A
budget of $20,000 could reach more than 40,000 people. Would you like to go ahead with
this strategy?
Feel Our recommended strategy uses a mix of short videos and static image formats. Our last
campaign with this creative mix cost $4,286 to reach 10,000 people. By contrast, a campaign
with longer videos cost almost twice as much, $8,333 to reach 10,000 people.

With Facebook IQ insights as evidence, stakeholders learn that video ads are a proven marketing
tool. The Conversion Lift results, communicated in business language, make stakeholders feel that
they’re getting value for the money they spend. The recommendation for the new campaign
communicates what the stakeholders need to do: Approve the campaign strategy and budget.

Test and learn mindset

Testing is both a science and an art, and it’s important to maintain a test-and-learn
mindset. With each test, you have an opportunity to keep testing further and generate new
insights to guide your marketing strategies.

There are two reasons why you might want to conduct another test. The first is to recheck
your hypothesis. To increase your confidence in the results or to account for unexpected
 conditions, you could conduct a similar test or repeat the same one to confirm that your
hypothesis holds. Secondly, you might want to use what you learned as the basis to test a
new hypothesis.

When to retest your hypothesis

There are several reasons to retest your hypothesis.

Inconclusive results
Your test results were inconclusive, which happens when there's no statistically
significant difference between test and control group results.

Contaminated results
Your test results could be contaminated, for example, if members of the control group are
exposed to the ad campaign.

Validate results
You need to confirm that your test results are valid. You can test the same hypothesis in a
different time frame or across a new audience segment.

Faulty measurement approach

Your measurement approach was faulty, for example, if your test design doesn't match the
hypothesis or you examined the wrong test metrics.

*****
The Storx marketing team runs a Facebook ad campaign to promote subscription boxes for
new parents. The team runs a single-cell Brand Lift test and gets 150 responses for each test
and control group. Lana, the data analyst, reviews the results:
 55% of test group respondents agree Storx is a leading brand for baby product subscription
boxes.
 50% of control group respondents agree Storx is a leading brand for baby product
subscription boxes.
 The results are statistically significant at the 50% confidence level.
What recommendation should Lana make from these insights?

Poll men and women separately.

Increase the target sample size.
Run a Conversion Lift test instead.
Run a multi-cell Brand Lift test.
The team should increase the target sample size. The Brand Lift test results are statistically
significant at the 50% confidence level. If the team want to increase their confidence in the test, they
need to increase the target sample size. This would increase the ability of the test to detect lift, which
results in a more powerful test.

The Storx team runs a multi-cell Conversion Lift test to compare an evergreen Facebook ad
campaign against a new dynamic ads campaign.
Their hypothesis states that the dynamic ads campaign will outperform the evergreen
campaign on conversion lift and sales lift. The test runs for six weeks.
The test results come back as follows:

 The dynamic ads campaign created a 4% lift in overall conversions and a $250,000
sales increase.
 The evergreen campaign created a 1.5% lift in overall conversions and a $100,000
sales increase.
 The dynamic ads campaign resulted in a $120.43 cost per incremental conversion and
a 1.5x ROAS.
 The evergreen campaign resulted in a $35.45 cost per incremental conversion and 2x
ROAS.

What’s the best recommendation that Lana can make from this information?
Continue to run both campaigns, but reduce spend on dynamic ads.
Reduce spend from both campaigns because they didn’t result in an efficient lift.
Shift the total budget to the dynamic ads campaign.
Shift the total budget to the evergreen campaign.
Dynamic ads created the highest additional sales and incremental conversions but also had the highest cost per
incremental conversion. The evergreen campaign had a significantly lower cost per incremental conversion
and a higher ROAS. The best recommendation is to continue both campaigns, reduce spend on dynamic ads.

Key takeaways
 Consider what you want stakeholders to know, feel and do when you communicate your campaign
recommendations.
 Adopt a test-and-learn mindset to generate new insights that can guide your marketing strategies.
 Retest your hypothesis to increase confidence in the results and to account for unexpected
conditions.
Sources
 Facebook IQ
 Experiment, Test and Learn