Week 8: Programming for Data Analysis
0 likes•259 views
This document discusses feature engineering techniques for data analysis. It covers feature selection, construction, and engineering. Specific techniques discussed include feature imputation, handling outliers, binning, log transforms, one-hot encoding, grouping, splitting, scaling, and extracting date features. The document provides examples and explanations of these techniques to transform raw data into more useful features for machine learning models.
Week 8: Programming for Data Analysis
- 1. Programming for Data Analysis Week 8 Dr. Ferdin Joe John Joseph Faculty of Information Technology Thai – Nichi Institute of Technology, Bangkok
- 2. Today’s lesson Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 2 • Feature Engineering • Feature Selection • Feature Construction • Laboratory
- 3. Importance Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 3
- 4. Features Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 4
- 5. Features Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 5
- 6. Features Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 6
- 7. Real Data features Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 7
- 8. Feature Engineering Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 8
- 9. Feature Engineering • Feature engineering is the process of using domain knowledge to extract features from raw data via data mining techniques. • These features can be used to improve the performance of machine learning algorithms. Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 9
- 10. Features • A feature is an attribute or property shared by all of the independent units on which analysis or prediction is to be done. Any attribute could be a feature, as long as it is useful to the model. • The purpose of a feature, other than being an attribute, would be much easier to understand in the context of a problem. A feature is a characteristic that might help when solving the problem. Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 10
- 11. Process of Feature Engineering Brainstorming or testing features Deciding what features to create Creating features Checking how the features work with your model Improving your features if needed Go back to brainstorming/creating more features until the work is done Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 11
- 12. Techniques in Feature Engineering • Imputation • Handling Outliers • Binning • Log Transform • One-Hot Encoding • Grouping Operations • Feature Split • Scaling • Extracting Date Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 12
- 13. Imputation • Missing values are one of the most common problems you can encounter when you try to prepare your data for machine learning. • The reason for the missing values might be human errors, interruptions in the data flow, privacy concerns, and so on. • This affects the performance of machine learning models Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 13
- 14. Imputation • Dropping columns with missing values will reduce performance • Make a threshold of 70% • Remove columns having more than 30% missing values Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 14
- 15. Numerical Imputation • Fill missing values with a constant • Fill missing values with a statistical formula Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 15
- 16. Categorical imputation • Replacing missing value with maximum occurred value in that column Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 16
- 17. Handling Outliers • Best way to detect outliers is to visualize data Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 17
- 18. Statistical ways to handle outliers • Standard Deviation • Percentiles Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 18
- 19. Handling outliers – Standard Deviation • If a value has a distance to the average higher than x * standard deviation, it can be assumed as an outlier. • x = 2 to 4 is practical. Z-score can also be used Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 19
- 20. Handling Outliers - Percentile • If your data ranges from 0 to 100, your top 5% is not the values between 96 and 100. • Top 5% means here the values that are out of the 95th percentile of data. Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 20
- 21. Binning • Binning is done for numerical data • Categorical data are converted to numerical format and binned Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 21
- 22. Binning - Example #Numerical Binning Example Value Bin 0-30 -> Low 31-70 -> Mid 71-100 -> High #Categorical Binning Example Value Bin Spain -> Europe Italy -> Europe Chile -> South America Brazil -> South America Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 22
- 23. Motivation of binning • Make the model robust • Prevent overfitting Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 23
- 24. Log Transform • Logarithmic Transformation • The data you apply log transform must have only positive values, otherwise you receive an error. • Also, you can add 1 to your data before transform it. • Thus, you ensure the output of the transformation to be positive. Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 24
- 25. Example Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 25
- 26. One hot encoding Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 26
- 27. Grouping Operations • Categorical Column Grouping • Numerical Column Grouping Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 27
- 28. Categorical Column Grouping Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 28
- 29. Numerical Column Grouping • Numerical columns are grouped using sum and mean functions in most of the cases. Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 29
- 30. Feature Split • Splitting features is a good way to make them useful in terms of machine learning. • By extracting the utilizable parts of a column into new features: • We enable machine learning algorithms to comprehend them. • Make possible to bin and group them. • Improve model performance by uncovering potential information. Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 30
- 31. Example Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 31
- 32. Scaling • In real life, it is nonsense to expect age and income columns to have the same range. • Scaling solves this problem. • However, the algorithms based on distance calculations such as k-NN or k-Means need to have scaled continuous features as model input. Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 32
- 33. Scaling Methods • Normalization • Standardization Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 33
- 34. Normalization • Normalization (or min-max normalization) scale all values in a fixed range between 0 and 1. • This transformation does not change the distribution of the feature and due to the decreased standard deviations, the effects of the outliers increases. • Therefore, before normalization, it is recommended to handle the outliers. Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 34
- 35. Normalization - Example Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 35
- 36. Standardization • Also known as z-score normalization • Scales the values while taking into account standard deviation. • If the standard deviation of features is different, their range also would differ from each other. • This reduces the effect of the outliers in the features. Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 36
- 37. Example Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 37
- 38. Extracting Date • Extracting the parts of the date into different columns: Year, month, day, etc. • Extracting the time period between the current date and columns in terms of years, months, days, etc. • Extracting some specific features from the date: Name of the weekday, Weekend or not, holiday or not, etc. Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 38
- 39. Extracting Date Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 39
- 40. DSA 207 – Feature Engineering Faculty of Information Technology, Thai - Nichi Institute of Technology, Bangkok 40