ENVR 5320
Environmental Data Analysis
Lecture 1
Dr. Zhi NING
Division of Environment and Sustainability
The Hong Kong University of Science and Technology
Agenda
• Environmental problems and statistics
• Brief review of statistics
• Get familiar with the Excel tools
• Statistical distribution measures
• Probability distributions
2
Environmental Problems and Statistics
• The goal of statistics
– to make discovery process efficient.
• Environmental laws and regulations:
– toxic chemicals;
– water quality criteria;
– air quality criteria.
• Environmental data
– the limit of detection;
– acute and chronic toxicity criteria;
– cancer potency factors.
3
Environmental Problems and Statistics
• Use statistic tools to understand the nature
4
Structure of course teaching
• Introduction
– engineering problem and statistical method.
• Case study
– introduce a specific environmental example with
real world data
• Method
– give a brief explanation of statistical method that is
used to prepare the solution.
• Analysis
– show how the data suggest and influence the
method of analysis and give the solution.
5
Brief Review of Statistics
• Population and sample
– A population is a very large set of N observations
(or data values) from which the sample of n
observations can be imagined to have come.
• Two types of statistics:
– DESCRIPTIVE
• a way of summarizing the complexity of data with a
single number.
– INFERENTIAL
• answer the question, "To what extent can these findings
be GENERALIZED?
6
Brief Review of Statistics
•DESCRIPTIVE Statistics
– For one variable ("univariate analysis"):
– Measures of "CENTRAL TENDENCY" (averages) and
of DISPERSION or variance around that average.
– Examples: Means, Modes, Medians, Standard
Deviation, quartiles etc
– For multiple variables:
– The strength of relationship between two variables
(bivariate analysis) or among a set of variables
(multivariate analysis)
– Examples: correlation coefficient
7
Brief Review of Statistics
•INFERENTIAL Statistics
– Measures of the SIGNIFICANCE of the relationship
between two or more variables. Significance refers to
the probability that the findings could be attributed to
sampling error.
– Appropriate statistics depend on the LEVEL OF
MEASUREMENT OF THE DEPENDENT VARIABLE
(and of the independent variable).
– Example: t-Test, ANOVA (F-ratio)
8
Let’s get Familiar with Excel Advanced
Tools
• Formula in Excel
• Hidden Developer functions in Excel.
• Practice calculations in Excel data example
• Good practice in using Excel
9
Excel basics I
• Use of formula
• Use of $
• Use of shortcut to go to cells
• Note the black and white cross
• Plot
• Use of Ctrl + Shift + Enter for array calculation
• Developer tool
• ActiveX
10
Statistical distribution measures
• Central values
– Arithmetic mean, Geometric mean
– Mode, Median
• Measures of spread
– The range
– The interquartile range (IQR)
– Standard deviation, variance
– Coefficient of variation (CoV)
• Quartiles, Quantiles and percentiles
11
Statistical distribution measures
• Central values
– Arithmetic mean Average(a,b,c)
– Geometric mean
–
Geomean(a,b,c)
– Mode: value with highest probability of occurrence
– The median: central value of the ordered data
Median(a,b,c)
• Trimmed mean:
– e.g. 5 percent trimmed mean is the average of the
data between 5th and 95th percentiles 12
Statistical distribution measures
• Influence of the shape of the data distribution
• “heavy tails”.
• Arithmetic mean is
• The “heaviness” of the
influenced by high
tails depends degrees of
values;
freedom (df)
• G is same as median
• G best represents
• Right skewed
• Higher df leads to
normal dist.
13
• Bimodal distribution in nature
• The implications
14
Statistical distribution measures
• Measures of spread
– The range (MIN and MAX)
– The interquartile range (IQR)
Percentile (array, k)
Quartile (array, 0/1/2/3/4)
IQR=0.7413*(Q3-Q1)
– The standard deviation
15
Statistical distribution measures
• Measures of spread
– Variance
VAR(array)
– Coefficient of variation (CV)
16
Statistical distribution measures
• Measures of spread
– Quartiles, quantiles and percentiles
Quartile (array, 0/1/2/3/4)
Percentile (array, 0.05/0.10/0.95)
– Skewness:
• measure of symmetry of data distribution
Skew (array)
0 is symmetric; <0, left skewed; >0, right
skewed.
17
Statistical distribution measures
• Frequency distributions
– Identify cutting points to divide the data into
categories. The cutoff points should be chosen to
divide the data fairly evenly.
Frequency (data_array,bin_array)
PRESS SHIFT/CTRL/ENTER
Bin Frequency
1 10 2
2 20 0
3 30 2
4 40 3
5 50 5
6 60 4
7 70 2
8 80 0
9 90 1
10 100 1 18
Statistical distribution measures
• Accuracy, Bias and Precision
– Bias measures systematic errors
– Precision measures the degree of scatter in the
data
– Accuracy is a function of both bias and precision.
A known concentration of 8.00 mg/L.
19
Probability distributions
• The Normal Distribution
– Often called Gaussian distribution
– Characterized completely by N(η, σ2 ), “a normal
distribution with mean η and variance σ2 .
20
Read and type Greek letters correction
• Alt 956
• Alt 963
• Alt 961
• Alt 960
https://www.thespruceeats.com/the-greek-
21
alphabet-1705558
Probability distributions
• The Normal Distribution
1. The vertical axis (probability density) is scaled
such that area under the curve is unity (1.0).
2. The standard deviation σ measures the distance
from the mean to the point of inflection.
3. The probability that a positive deviation from the
mean will exceed one σ is 0.1587.
4. Because of symmetry, the probabilities are the
same for negative deviations
5. The chance that a deviation in either direction will
exceed 2σ is 2(0.0228) = 0.0456
22
Probability distributions
• NORM.DIST(x, mean, standard_dev, cumulative)
– Returns the normal cumulative distribution of with specific η and σ.
– Returns α value with given z and η σ values.
• NORM.INV (probability, mean, standard_dev)
– Returns the inverse of the normal cumulative distribution for η and σ.
– Returns z value with given α, η and σ values
• NORM.S.DIST (z, cumulative)
– Returns the standard normal cumulative distribution of with η=0 and σ=1
• NORM.S.INV (probability)
– Returns the inverse of the standard normal distribution with η=0 and σ=1
• Cumulative or not?
• Left tailed or right tailed?
• How to generate a normal distribution in excel?
23
Probability distributions
• Examples
– A normal distribution with η=8mg/L and σ=1 mg/L;
– Look for the value with 95% of data below?
– Look for the probability that the value is read
below 6.4mg/L?
– How to draw a normal distribution in Excel?
– Use function: norm.inv(rand(),8,1,1)
24
Probability distributions
• t distribution
– In normal distributions, both η and σ are known;
– In practice, σ is often not known and we use Se to
replace σ:
– Bell shaped and symmetric but tails are wider.
– Width of the t distribution depends on degree of
freedom.
Guinness brewer
Gosset, 1908 25
“Student” as pen name
Probability distributions
• Part of the t table as function of and
26
Probability distributions
• T.INV (probability, degree of freedom)
– Returns the inverse of the left tailed Student t distribution
• T.INV.2T (probability, degree of freedom)
– Returns the inverse of the two tailed Student t distribution
• T.DIST (x, degree of freedom, cumulative)
– Returns the left tailed Student t distribution
• T.DIST.RT (x, degree of freedom, cumulative)
– Returns the right tailed Student t distribution
• T.DIST.2T (x, degree of freedom, cumulative)
– Returns the two tailed Student t distribution
• If we enter α as probability and n-1 as Deg_freedom, then T.INV
outputs tn-1, 1-α/2, the 1-α/2 th percentile of a t distribution with n-1
degrees of freedom.
27
Probability distributions
• Example
– What is the 97.5th percentile of a t distribution with
degree of freedom 24 ?
– T.INV.2T(0.05, 24)=2.06
OR -T.INV(0.025,24)
– What is the probability of t value larger than 2.064
in a t distribution with degree of freedom 24?
– T.DIST.2T(2.064,24)
28
Distribution of average and variance
• Consider a sampling distribution of the
average, with many random samples of size n
were collected from a population
• Sample standard deviation:
• Standard error of the mean is:
29
Distribution of average and variance
• Central limit effect:
– If parent distribution where the samples come
from is normal, the distribution of average is
normal
– If the parent distribution is not normal, the
distribution of average will be more nearly normal
than the parent one.
– With increasing number of sample n, the
distribution becomes increasingly more normal.
30
Distribution of average and variance
• How to estimate the t statistic?
– From normal parent population to samples with t
distribution with df= n-1:
– The sample variance s2 is distributed as Chi-
square distribution:
31
Distribution of average and variance
• Example:
From Sd to Se, 0.266
NORM.DIST(7.51,8,0.27,1) With t=-1.842 and =26,
T.DIST(-1.842,26,1)
N(8,0.27)
32
Tutorial session
