Scribd
Upload
7 views18 pages

Lecture 2.3.11

Uploaded by

hw941885
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
7 views18 pages

Lecture 2.3.11

Uploaded by

hw941885
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 18

APEX INSTITUTE OF TECHNOLOGY

(CSE)
DEPARTMENT OF COMPUTER SCIENCE & ENGINEERING

Predictive Analytics (21CSH-340)


Faculty: Dr. Jitender Kaushal
Associate Professor
(E14621)
DISCOVER . LEARN . EMPOWER
SQL Server – How to handle NULL data 1
COURSE OBJECTIVES
Course Objectives
 Understand how analytics provided a solution to industries using real case
studies.
 Explain modeling, relationships, derive and reclassify fields, integrate and collect
data.
 Review and explore data to model.

2
COURSE OUTCOMES
On completion of this course, the students shall be
able to

Unit-2 To perform data collection and initial data handling by CO3


managing data structures and measurement levels.
Analyzing and transforming data for predictive modeling CO4
through various data transformation techniques.

3
The Problem with NULL
 There is a problem with NULL that has persisted since the
Relational
Model was proposed in the 1970’s.
 “The simple scientific fact is that an SQL table that contains a null isn’t a relation; thus,
relational theory doesn’t apply, and all bets are off. ” C.Date (2014).

 The presence of NULLs in a database ‘breaks’ the relational


model of
Boolean expressions on which SQL databases rely.

 In ‘real world’ applications of data structures NULLs are


often unavoidable.

 It confuses users, and designers and DBA’s hate it.


Three Valued Logic
 The SQL language is based on Relational Logic.

 Adding NULL Values to a database breaks the TRUE/ FALSE relations


implicit in the
model and leads to ‘TRUE’, ‘FALSE’ and ‘UNKNOWN’

 At best this leads to increased complexity by having to use horizontally

decomposed WHERE clauses, workaround syntax and inference.

 At worst leads to incomplete information, returned error codes,

interoperability problems, interpretation problems.

 Messes up Reporting, ETL, Business Intelligence and Data Science


initiatives.
Domain Knowledge
 SQL Databases are modelled as domains, and as such the designer

needs to be able to define what the domain encompasses by


defining the boundaries, identifying components and relationships.

 Almost by definition the designer will have incomplete

information about the information that is relevant, especially


when implementing new systems.

 NULL is stored as a flag, therefore is not part of any particular

domain or type and in making assumptions about NULLs is where


query complexity is introduced.
Null is not a value, it is not zero, it is unknown
NULL Data Scenarios
 Existence -Attribute does not exist in the domain, or domain understanding is
wrong. E.g eye
colour for a car.

 Missing – The information has not been given at the time a row was created. E.g. A

customer my decline to give their age.

 Not Yet – Data is contingent upon an unknown event in the future, E.g. Termination

date or Date of death.

 Does not apply- Is not applicable for this instance of a record . E.g. Hair colour for
bald people,
Number of pregnancies for male patients.

 Placeholders – Indicates that we know that a bit of data exists, but we don’t know
Handling NULL in Queries
 NULLIF
 Syntax: NULLIF (expression, expression)
 Returns NULL if both expressions are equal, else returns the first expression.

 ISNULL to check the state of a field


 Syntax: ISNULL (check_expression, replacement_value)
 Returns replacement value that must be implicitly convertible to check
expression data type.

 COALESCE to use the first non-null field.


 Syntax: COALESCE( exp1, exp2,…expn)
 Can use multiple input expressions.
 Returns the datatype of the expression with highest precedence.
 Slower than ISNULL.
Handling NULL in WHERE
Clauses
 Using Three Value Logic (True,False,Unknown). UNKNOWN is the logical
outcome and is
not the same as NULL.

 To compare values we have to use the IS NULL and IS NOT NULL operators in
the
WHERE clause, not the = operator.

 IS NULL
 SELECT * From Customers WHERE CustName IS NULL

 IS NOT NULL
 SELECT * From Customers WHERE CustName IS NOT NULL

 Use Horizontal Decomposition to add other conditionals



Environment and
Aggregate Settings

 ANSI_NULLS environment setting.


 When creating or altering stored procedures or User Defined Functions.
 This option specifies the setting for ANSI NULL comparisons. When this is on, any
query that compares a value with a null returns a 0. When off, any query that
compares a value with a null returns a null value.
 Keep at default value of ON.

 “Null value is eliminated by an aggregate or other SET


operation”
 Trying to do arithmetic or other operation on fields that contain NULLS.
 May lead to incomplete information returned.
 Use ISNULL or NULLIF to prevent this happening.
Table Design
Guidelines
 Design Integrity into your tables.

 Use NOT NULL CHECK() constraints where possible.

 Do not use as a Primary key if there is ANY possibility

that value could be NULL.


 Avoid in FOREIGN KEY relationships
 Consider using de-normalised separate tables to get around this.

 Use default field values where appropriate.


 Bear in mind arithmetic consequences of using 0, -99 as defaults.
App Design Guidelines
 Take steps to avoid NULL values from host
programs.
 Initialisation of variables
 Use defaults and appropriate auto filling of variable
values
 Deduce values
 Track missing data using companion codes
 Determine the impact of missing data
 Validate data and prevent audit difficulties
 Use consistent datatypes and nullability across apps
ETL Guidelines
 Where multiple fields may contain NULL, consider using

a check code field to indicate where records need


attention.
 Check the NULL status of each field using ISNULL

(Field,0) and build a count of the number of fields that


fail validation.
 Use this as part of the data cleansing process.

 Use in ETL and Scrubbing tables


Master Data Management
Management is about catching the information before it enters the
database and
cleaning up what is already there.
 MDS
 Master Data Services is included as part of SQL Server.
 Allows Models, Entities, Attributes, Rules and Versions to be defined and
implemented.
 Includes Excel add in. Allows Power users or analysts to define models and
rules.

 DQS
 To help ensure domain validity and knowledge driven data quality.
 Good for data correction, enrichment, standardization, and de-duplication.

 Other third party applications available


 Master Data Maestro, etc.
Performance Considerations
Time spent in designing appropriate data quality controls
will reduce the cost of maintaining the database because

 NULL slows down the working of indexes.


 Increases query retrieval times.
 Increase search times.
 Increases SQL code complexity.
 Decreases the confidence in the information gained
from the database.
Data Science
NULLS may provide the catalyst for development of data science to
discover why,
or what data or domain knowledge we are lacking.

 Null indicates value is not known or indicates missing or incomplete


data.

 May point to missing entities or uncaptured events.

 May skew the results of data tools that disregard NULL values.

 Known knowns, Unknown knowns and Unknown unknowns. Data

discovery and Knowledge begin by examining what it is that is


Summary
 The presence of NULL values in SQL databases has always happened, but this
degrades the
quality of the information that can be obtained from the data source.

 NULLs can have an adverse effect on downstream systems, in particular

Reporting, BI, Predictive Analytics or Machine Learning that rely on the


integrity of the data.

 Reduce the impact on your information by:


 Manage the quality of data going in
 Design tables with integrity constraints
 Design apps to validate the input
 Design queries to ensure correct results are returned

 Use NULLs as clues to pick up where domain knowledge is lacking.


THANK YOU

For queries
Email: [email protected]

18

You might also like