Scribd
Upload
105 views2 pages

Data Analytics 02: Drag Connect It Change Remove Cabin, Life Boat, Name, and Ticket Number

1. The document discusses identifying and removing outliers from a Titanic dataset for improved data quality. It describes normalizing the data, detecting outliers using a distance-based algorithm, and filtering outliers from the dataset. 2. Key steps include normalizing the data to put attributes on the same scale, using an operator to detect the 10 farthest outliers and mark them, and filtering to remove outliers from the final dataset. 3. The document provides guidance on using breakpoints to view intermediate results and instructions for modifying the process to find more outliers or only show outliers without removing them.

Uploaded by

Jhon Rey Balbastro
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
105 views2 pages

Data Analytics 02: Drag Connect It Change Remove Cabin, Life Boat, Name, and Ticket Number

1. The document discusses identifying and removing outliers from a Titanic dataset for improved data quality. It describes normalizing the data, detecting outliers using a distance-based algorithm, and filtering outliers from the dataset. 2. Key steps include normalizing the data to put attributes on the same scale, using an operator to detect the 10 farthest outliers and mark them, and filtering to remove outliers from the final dataset. 3. The document provides guidance on using breakpoints to view intermediate results and instructions for modifying the process to find more outliers or only show outliers without removing them.

Uploaded by

Jhon Rey Balbastro
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 2

DATA ANALYTICS 02

IDENTIFY ANOMALIES AND OUTLIERS IN YOUR DATA.


Another important step of data cleansing is to identify unusual cases and remove them from the
data set. In some situations, the outliers themselves might be the most interesting cases
(detecting fraudulent credit card transactions, for example), but in most cases outliers are simply
the result of an incorrect measurement and should be removed from the data set. This is exactly
what we will do in this lecture.
PREPARE THE DATA.

1. Drag the Titanic data into the process.


2. Add the operator Select Attributes and connect it.
3. Change the Parameters so that you remove Cabin, Life Boat, Name, and Ticket Number .

NOTE: The result will be a data set only containing those columns we believe will contribute well
to our outlier detection. We will use a distance-based outlier detection algorithm which
calculates the Euclidean distance between the data points and marks those points which are
farthest away from other data points as outliers. The Euclidean distance uses the distances
between two data points for each individual attribute. Think about it: what is the effect on the
distance if the attributes have different value ranges (one attribute between 0 and 5 and
another attribute between 1 and 1000)? Attributes with larger values will contribute much more
than those with smaller values. For this reason we should ensure that all attributes are using
similar value ranges. This transformation is called Normalization.

NORMALIZE THE ATTRIBUTE VALUE RANGES.

1. Add the operator Normalize and connect it.

NOTE: In general, you should always normalize your data before you apply distance-based
algorithms like outlier detection or k-Means clustering. Using the default parameters, the
Normalize operator will perform a z-Transformation (also known as Standardization) which
results in a mean value of 0 and a standard deviation of 1 for each attribute. In other words, all
of the attributes are on the same scale after normalization and can be compared with one
another.

DETECT OUTLIERS.

1. Search for the operator Detect Outlier (Distances), add it, and connect it to Normalize.

NOTE: This operator will identify the 10 examples which are farthest away from all others and
mark them as outliers. It creates a new column named outlier with true as the value for the
outliers and false for all other examples.

Page 1 of 2
Dr. Stephan Kupsch
DATA ANALYTICS 02

REMOVE OUTLIERS FROM EXAMPLE SET.

1. Add Filter Examples to the process and connect it to the previous operator and also to
the result port on the right.
2. In its Parameters, add a new filter with Outlier, equals, and false as values.

3. Run the process.

NOTE: The process might run for some time but will switch to the Results view automatically
when it is finished. You will notice that the result is a data set with 1299 examples - the 10
outliers have successfully been removed.
USE BREAKPOINTS TO SEE INTERMEDIATE RESULTS.

1. Go back to the Design view.


2. Right-click on Detect Outlier (Distances) and select Breakpoint after in the menu. Note
that a small icon appears at the bottom of the operator after you added the breakpoint.

3. Run the process again.

NOTE: The process is paused at the breakpoint and the intermediate results are shown. This
makes breakpoints a useful tool for debugging your process! All 1309 examples are still in the
data set at this point of the process. Click the Resume icon in the toolbar to continue the process
and see the final result.
By now, you found 10 outliers in your data and sucessfully removed them! This cleansing step
can often improve the quality of your models.

TASKS:

 How would you change the process so it finds 20 outliers instead of 10?
 How can you change the process to only show outliers instead of removing them?
 Replace the outlier detection operator with Detect Outlier (LOF) and add a breakpoint
after this operator before you execute. What is the difference to before?
 How do you now need to change the filter to only keep the top outliers?

Page 2 of 2
Dr. Stephan Kupsch

You might also like