TONGJI UNIVERSITY

Missing data solution of electricity consumption based on Newton’s Divided-Difference

Interpolation Case Study: IntelligEnSia data monitoring

学 院：土木工程学院
系 所：桥梁工程系
姓 名：杜别 2493252
指导老师：陈艾荣

2024 年 12 月 11 日
 Table of contents

Abstract
Missing data or values is a common issue in processing a dataset. It is also occurred in the
IntelligEnSia system, which is a system that utilizes and optimizes the electricity consumption
data. The problems occur when the data that are being sent by the sensor(s) to the web server
are missing due to the unstable internet connection. It is an essential matter, since we want to
capture the data by real time. The data set are useful to learn the pattern of the electricity
consumption and predict the next electricity demand. Therefore, to overcome these problems
we try to propose a method to complete the missing data by applying Newton’s divided-
differences interpolation method. So, this research may support to predict the electricity
consumption and to create an effective prediction model by solving the problem of missing
data.

Introduction
Missing data is identically referred to a missing data mechanism, observation process or
selection process. In context of missing data process, it is lead to a missing data theory that
has an impact on learning, inference and prediction. There are two fundamental categories of
missing data: random and un-random missing data. In case of random missing data, the
process can be ignored and inference can be based only by the observed data. In this case, in
many common generative models the computing results are traceable. For the un-random
missing data, the missing one will be ignored. Missing data in the data set used for any data
analysis is serious issue for any research. There are various methods that are used for finding
the missing data in data set and these are having different conditions for finding missing data
from these methods some of methods are for handling the missing data and some of methods
are for interpolating or predicting the missing value in data set (Manembu, Kewo et al. 2015).
In this paper is on how to solve electricity consumption missing data based on IntelligEnSia
using Newton’s divided-differences interpolation method. Then objective is to interpolate
missing data in the electricity consumption based on the observed data from IntelligEnSia
using Newton’s divided-differences interpolation method.

I. Background
Numerical analysis is a branch of mathematics that solves continuous problems using numeric
approximation. It involves designing methods that give approximate but accurate numeric
solutions, which is useful in cases where the exact solution is impossible or prohibitively
expensive to calculate. Numerical analysis also involves characterizing the convergence,
accuracy, stability, and computational complexity of these methods. (Chapra 2011).
The numerical analysis method used to solve the problem of this paper is: Newton’s Divided-
differences interpolation method. It is among the most popular and useful method used for
the interpolations.
 I.1. Newton’s divided-difference interpolation method
Newton's divided-differences interpolation is based on the concept of finite differences and
stems from a reformulation of the Lagrange interpolation polynomial into a more modular and
computationally efficient form. This theorem states that for a set of distinct points, there exists
a unique polynomial of degree at most that passes through all these points. Newton proposed
a constructive method to express this polynomial using divided differences, which measure
the variations in -values between the -points. These Divided differences are recursively
calculated coefficients representing the average rate of change between multiple points. It is
used to compute the coefficients of the interpolation polynomial. The polynomial is expressed
in a hierarchical structure, with successive terms based on divided differences. This form
relies on the modular structure of the products, making computations efficient and allowing
for easy addition of new data points (Chapra 2018).
Newton's method is mathematically better to Lagrange interpolation in terms of the resulting
polynomial and it is more practical for:
 Incremental computations (adding a new point does not require recalculating the entire
polynomial).
 Handling unequally spaced data points.
I.2. Newton’s divided-difference interpolating polynomial
Newton’s divided-difference interpolating polynomial is among the most popular and useful
forms. Before presenting the general equation, we will introduce the first and second-order
versions because of their simple visual interpretation (Burden and Faires 2010).
I.2.1. Linear Interpolation
The simplest form of interpolation is to connect two data points with a straight line. This
technique, called linear interpolation, is depicted graphically in Fig. 1. Using similar triangle
f 1 ( x )−f ( x 0 ) f ( x 1 )−f ( x 0 )
=
x −x 0 x 1−x 0
( 1)
which can be rearranged to yield
f ( x 1) −f ( x 0 )
f 1 ( x )=f ( x 0 ) + ( x−x 0 )
x 1−x 0
( 2)
which is a linear-interpolation formula. The notation f 1 ( x ) designates that this is a first order
interpolating polynomial. Notice that besides representing the slope of the line connecting the
points, the term [ f 1 ( x )−f ( x 0 ) ] /( x−x 0) is a finite-divided-difference approximation of the first
derivative. In general, the smaller the interval between the data points, the better the
approximation. This is due to the fact that, as the interval decreases, a continuous function
will be better approximated by a straight line.
 Figure 1 : Graphical depiction of linear interpolation. The shaded areas indicate the similar triangles used in Eq.2

I.2.2. Quadratic Interpolation

The errors caused from some approximations is due to use a straight line to approximate a
curve. Consequently, a strategy for improving the estimate is to introduce some curvature into
the line connecting the points. If three data points are available, this can be accomplished with
a second-order polynomial (also called a quadratic polynomial or a parabola). A particularly
convenient form for this purpose is (Chapra 2018):
f 2 ( x )=b 0 +b1 ( x−x 0 ) +b 2 ( x−x 0 ) ( x−x 1 )
( 3)
or, collecting terms,
2
f 2 ( x )=a0 +a1 x+ a2 x

Where
a 0=b 0−b1 x 0+ b2 x0 x 1

a 1=b1−b2 x 0 −b2 x 1

a 2=b2

A simple procedure can be used to determine the values of the coefficients. For b 0, Eq.3 with
x−x 0 can be used to compute

b 0=f ( x 0 )
(4 )
Eq. 4 can be substituted into Eq.3, which can be evaluated at x−x 1 for
f ( x 1) −f ( x 0 )
b 1=
x 1−x 0
(5)
Finally, Eq. 4 and 5 can be substituted into Eq. 3, which can be evaluated at x=x 2 and solved
(after some algebraic manipulations) for x−x 2
f ( x 2 )−f ( x 1 ) f ( x 1) −f ( x 0 )
−
x2− x1 x 1−x 0
b 2=
x 2−x 0
(6)
Notice that, as was the case with linear interpolation, b 1 still represents the slope of the line
connecting points x 0 and x 1. Thus, the first two terms of Eq. 3 are equivalent to linear
interpolation from x 0 to x 1., as specified previously in Eq. 2. The last term, b 2 ( x−x 0 ) ( x−x 1 ),
introduces the second-order curvature into the formula.
I.2.3. General Form of Newton’s Divided-difference Interpolating Polynomials
The preceding analysis can be generalized to fit an nth-order polynomial to n + 1 data points.
The nth-order polynomial is (Chapra 2018):
f n ( x )=b0 +b 1 ( x−x 0 ) +…+ bn ( x−x 0 ) ( x−x 1 ) … ( x−x n−1)
(7 )
As was done previously with the linear and quadratic interpolations, data points can be used
to evaluate the coefficients b 0, b 1, . . . , b n. For an n th-order polynomial, n+1 data points are
required: [ x 0, f ( x 0 ) ], [ x 1, f ( x 1 )], . . . , [ x n , f ( x n )]. We use these data points and the following
equations to evaluate the coefficients:
b 0=f ( x 0 )
(8 )
b 1=f [ x 1 , x 0 ]
(9 )
b 2=f [ x 2 , x 1 , x 0 ]
( 10 )
.
.
.
b n=f [ x n , x n−1 , … , x 0 ]
( 11)
where the bracketed function evaluations are finite divided differences. For example, the first
finite divided difference is represented generally as
f ( x i )−f ( x j )
f [ x i , x j ]=
x i−x j
( 12 )
The second finite divided difference, which represents the difference of two first divided
differences, is expressed generally as
f [x i , x j ]−f [ x j , x k ]
f [ x i , x j , x k ]=
x i−x k
( 13 )
f [ xn , x n−1 , … , x 0 ]−f [ x n−1 , x n−2 , … , x 0 ]
f [ x n , xn −1 , … , x0 ]=
x n−x 0
( 14 )

Figure 2 : Graphical depiction of the recursive nature of finite divided differences (Chapra 2018).

These differences can be used to evaluate the coefficients in Eq. 8 through 11, which can then
be substituted into Eq. 7 to yield the interpolating polynomial
f n ( x )=f ( x 0 ) + ( x−x 0 ) f [ x 1 , x 0 ] + ( x−x 0 ) ( x−x 1 ) f [ x 2 , x 1 , x 0 ] +…

+ ( x−x 0 ) ( x−x 1 ) … ( x−x n−1 ) f [ x n , x n−1 ,… , x 0 ]

(15 )
which is called Newton’s divided-difference interpolating polynomial.

II. IntelligEnSia
IntelligEnSia is an Information Communication and Technology (ICT) innovative solution
that can control and monitor the energy consumption through the internet connection on web
and mobile platforms. It can also adapt and learn the user consumption profile. Based on the
related data, IntelligEnSia can produce intelligence flexible reporting. IntelligEnSia’s
application: Monitoring IntelligEnSia optimized the data by providing a flexible reporting for
the user. It contains the information of Voltages, current and power which are the basic
variables in measuring the energy consumption. Voltage and current can be captured by the
voltage and current sensors that have been plug in at home or building. While Power, is the
multiplication result of voltage and current (Manembu, Kewo et al. 2015).
P=I .V
( 16 )
Where P : power (watt), I : current (Ampere), V : voltage (volt)
Experiments and results
IntelligEnSia Data Monitoring
In the data monitoring, as a result of unstable internet connection we find out some missing
data as captured on the www.intelligensia.org:
 V o l t a ge (V o l t )

226 226 222 222

220 220

0
17:40:00 17:40:30 17:41:00 17:41:30 17:42:00 17:42:30 17:43:00

Figure 3 : Curve of voltage from IntelligEnSia (Manembu, Kewo et al. 2015)

C u rren t (Amp ere)

2.592 2.592

2.187 2.187

1.782
1.692

0
17:40:00 17:40:30 17:41:00 17:41:30 17:42:00 17:42:30 17:43:00

Figure 4 : Curve of current from IntelligEnSia (Manembu, Kewo et al. 2015)

P o w e r (Wa tt )
570.24 570.24

494.262 494.262

395.604
375.624

0
17:40:00 17:40:30 17:41:00 17:41:30 17:42:00 17:42:30 17:43:00

Figure 5 : Curve of power from IntelligEnSia (Manembu, Kewo et al. 2015)

To obtain the energy consumption data in a house or building, IntelligEnSia captures real time
data with the interval of 30 seconds. It may benefit to the customer who wants to know how
much power that he/she has used by real time. In the real time capturing process, sometimes
the data are unsuccessfully captured by the IntelligEnSia server due to the down of electricity
by the provider or unstable internet connection. In this case, if server cannot capture the
voltage and current data, it is then automatically cannot count the power. Our focus is on the
missing data when the unstable internet connection is occurred. Table below displays that on
17:42:00, the data of Voltage, Current and Power are valued null, which may affect the power
measurement result (Manembu, Kewo et al. 2015).
Table 1 : The example of missing data in IntelligEnSia (Manembu, Kewo et al. 2015)

Voltag Current Power

Nr. Time
e (Volt) (Ampere) (Watt)
1 17:40:00 226 2.187 494.262
2 17:40:30 226 2.187 494.262
3 17:41:00 220 2.592 570.24
 4 17:41:30 220 2.592 570.24
5 17:42:00 0 0 0
6 17:42:30 222 1.692 375.624
7 17:43:00 222 1.782 395.604

III. Methodology
Newton’s divided-difference Interpolation is the interpolation technique that using the
polynomial functions, so there are some required steps:
 Selection of the origin of your coordinate system:
To get the idea of interpolation or missing value in the set data, it’s necessary de plot the set
data in the graph. The origin of this coordinate system is very important to plot the graph. It
permits also to get the first point of your set data(x 0 ; f ( x 0 ) )

 Plot the set data in the coordinate system:

To calculate the polynomial function of Newton’s divided-difference method, we need the
values of x 0 , x 1 , … , x n and its corresponding values f ( x 0 ) , f ( x 1 ) , … , f ( x n ). For that, select the
times set data as the x values and the corresponding values of each time set data in the current
or the voltage as the corresponding f (x) values.
 Plot a table for each point:
The interval of recording is 30 second, it’s possible to use only this unit to make the time table
for each point of the set data. Example: let’s take the first recording data as the origin of our
coordinate system. So, for x 0=0 , we have for the table above voltage=226Volt ,
current =2.187 Ampere and power=I . V =494.262 Watts. After this procedure, we will have
this kind of table for each point like this
Table 2 : The example of missing data in IntelligEnSia converted in coordinate system points

Positi y voltage ( x) y current ( x) y power (x)

on
x (s) (Volt) (Ampere) (Watt)

0 226 2.187 494.262

30 226 2.187 494.262
60 223 2.592 570.24
90 220 2.592 570.24
120 0 0 0
180 222 1.692 375.624
210 222 1.782 395.604
  Selection of the order the interpolation polynomial:
The order of the interpolation method depends on the set data that are given. It’s equal to the
number of the completed data (without missing data) mines one. For the above example:
n=6−1=5
  Calculation:
We can directly use all these information in the Python or MATLAB code to interpolate the
missing value of the voltage and the current. It’s also possible to use all these codes to
calculate all the coefficients and write the interpolation polynomials and then use it to calcite
the missing data at x=120 seconds

IV. Calculation
More the order is high, more we need some software or some codes to calculate the missing
data. It’s simple and rapid. For this paper, I used the Python and MATLAB codes to solve this
problem. And this part is only focus on how to make a code for this calculation.
The codes (Python or MATLAB) are based on the recursive property of this method shown in
the figure below:
Table 3 : recursive graphic of Newton’s divided-differences interpolation method (Burden and Faires 2010)

 Define an array for the values of x and y=f (x ) as shown above;

 Define a table P in which, we will have all coefficients for the interpolation
polynomials including also the values of x and y=f (x ). This array will have two
dimensions (rows and columns);
 Make some calculus in the array P to get the coefficient for the interpolation
polynomial according to the formula of these coefficients;
 Use the command lst [ ] to calculate the product ( x−x 0 ) ( x−x 1 ) … ( x−x n−1 ) for each
member of the function.
 Enter the position of the missing value x and then run.
According to these steps with some requirements for scripting, we will have the following
codes that give us the coefficients, the interpolate value at the point x
 IV.1. Python code
After entering all of the values of x and y=f (x ), the code will ask you to input the position of
the point that need to be interpolate.
## Newton Divided Difference Polynomial Interpolation Method

import numpy as np

x=np.array([ ],float) # values of x

y=np.array([ ],float) # values of y=f(x)
n=len(x)
p=np.zeros([n,n+1])#creating a Tree table (n x n+1 array)
value =float(input("Enter the point at which you want to calculate the value of the polynomial: "))
# first two columns of the table are filled with x and y data points
for i in range(n):

p[i,0]=x[i]
p[i,1]=y[i]

## algorithm for tree table from column 2 two n+1

for i in range(2,n+1): #column
for j in range(n+1-i):# defines row
p[j,i]=(p[j+1,i-1]-p[j,i-1])/(x[j+i-1]-x[j])#Tree Table
np.set_printoptions(suppress=True)## this suppress the scientific symbol(e) and returns values in normal
digits

# print(p) ## can check the complete Tree table here for NDDP
b=p[0][1:]#This vector contains the unknown coefficients in the polynomial which are the top elements of
each column.
print("b= ",b)
print("x= ",x)
lst=[] # list where we will append the values of prouct terms

t=1
for i in range(len(x)):
t*=(value-x[i]) ##(x-x0), (x-x0)(x-x1), (x-x0)(x-x1)(x-x2) etc..
lst.append(t)
print("The list of product elements ",lst,end = ' ')
## creating a general function
f=b[0]
for k in range(1,len(b)):
f+=b[k]*lst[k-1] ## important**[k-1]** not k because in list we use one step earlier element. For example
for b1 we have to use (x-x0), for b2, we use (x-x0)(x-x1) for b3 we use (x-x0)(x-x1)(x2)
print("The value of polynomial: ","%.3f"%f)
 IV.2. MATLAB code
It’s the same procedure like Python
% Newton's Divided Difference Interpolation Method

% Input: Data points (x, y)

x=[ ]; % values of x
y=[ ]; % values of y=f(x)

% Check if the input sizes match

if length(x) ~= length(y)
error('The length of x and y vectors must be the same.');
end

% Number of data points

n = length(x);

% Construct the divided difference table

div_diff = zeros(n, n);
div_diff(:, 1) = y'; % First column is y values

for j = 2:n
for i = 1:n-j+1
div_diff(i, j) = (div_diff(i+1, j-1) - div_diff(i, j-1)) / (x(i+j-1) - x(i));
end
end

% Extract the coefficients of the polynomial

b = div_diff(1, :); % Coefficients of the polynomial
disp('Coefficients of the interpolation polynomial (b):');
disp(b);

% Input: Value to interpolate

xi = input('Enter the value to interpolate: ');

% Compute the interpolated value

interpolated_value = b(1); % Initialize with the first coefficient
product_term = 1; % For (xi - x0)(xi - x1)...
for i = 2:n
product_term = product_term * (xi - x(i-1));
interpolated_value = interpolated_value + b(i) * product_term;
end

% Display the interpolated value

disp('The interpolated value is:');
disp(interpolated_value);

V. Results
For this part, let’s use the set data for the table 2 and apply the formula in Python, we will get
at x=120 seconds (missing point):
 Current I =2.192 Ampere
 Voltage U =218.829Volts
 Power P=U . I =479.673168Watts
So, with the missing value obtained, we will get the table and the figures below
 Table 4 : Correction of the set data of the table 1

Voltage Current Power

Nr. Time
(Volt) (Ampere) (Watt)
1 17:40:00 226 2.187 494.262
2 17:40:30 226 2.187 494.262
3 17:41:00 220 2.592 570.24
4 17:41:30 220 2.592 570.24
5 17:42:00 218.829 2.192 479.673
6 17:42:30 222 1.692 375.624
7 17:43:00 222 1.782 395.604

Voltage (Volt)
226 226

222 222
220 220
218.829

17:40:00 17:40:30 17:41:00 17:41:30 17:42:00 17:42:30 17:43:00

Figure 6 : Voltage curve corrected

Current (Ampere)
2.592 2.592
2.187 2.187 2.192
1.692 1.782

17:40:00 17:40:30 17:41:00 17:41:30 17:42:00 17:42:30 17:43:00

Figure 7 : Current curve corrected

 Power (Watt )
570.24 570.24
494.262 494.262 479.673

375.624 395.604

17:40:00 17:40:30 17:41:00 17:41:30 17:42:00 17:42:30 17:43:00

Figure 8 : Power curve corrected

Discussions and Conclusion

Interpolation, which is a tool for estimating the value of the dependent variable corresponding
to a value of the independent variable lying between its two extreme values on the basis of the
given values of the independent and the dependent variables. A number of interpolation
methods such as Newton’s Forward Interpolation formula, Newton’s Backward Interpolation
formula, Lagrange’s Interpolation formula, Newton’s Divided Difference Interpolation
formula, Newton’s Central Difference Interpolation formula, Stirring’s formula, Bessel's
formula and some others are available in the literature of numerical analysis (Thant).
However, a formula is necessary for representing a given set of numerical data on a pair of
variables by a suitable polynomial. The polynomial that represents the given set of numerical
data can be used for interpolation at any position of the independent variable lying within its
two extreme values (Das and Chakrabarty 2016).
The approach of interpolation, described here, can be suitably applied in inverse interpolation
also. Newton’s forward interpolation formula is valid for estimating the value of the
dependent variable under the following two conditions (Das and Chakrabarty 2016):
 The given values of the independent variable are at equal interval.
 The value of the independent variable corresponding to which the value of the
dependent variable is to be estimated lies in the first half of the series of the given
values of the independent variable.
However, Newton’s divided difference interpolation formula is valid for estimating the value
of the dependent variable beyond these two conditions. Therefore, the formula derived here is
valid for representing a set of numerical data on a pair of variables by a polynomial beyond
these two conditions. It involves fewer arithmetic operations than Lagrange’s form.
Moreover, it often hap-pens that we have to increase the degree n to reach a required
accuracy. the further increase of the polynomial function degree doesn’t need to restart the
interpolation. It’s very easy to add some points and continue the interpolation (Thant).
 References
Burden, R. L. and J. D. Faires (2010). Numerical analysis, Brooks Cole.
Chapra, S. (2011). EBOOK: Applied Numerical Methods with MATLAB for Engineers and
Scientists, McGraw Hill.
Chapra, S. C. (2018). Applied numerical methods with MATLAB for engineers and scientists,
Mcgraw-hill.
Das, B. and D. Chakrabarty (2016). "Newton’s divided difference interpolation formula:
Representation of numerical data by a polynomial curve." International Journal of
Mathematics Trends and Technology-IJMTT 35.
Manembu, P., et al. (2015). Missing data solution of electricity consumption based on
Lagrange Interpolation case study: IntelligEnSia data monitoring. 2015 International
Conference on Electrical Engineering and Informatics (ICEEI), IEEE.
Thant, A. A. Newton's Divided Difference Interpolation to Robot Path Planning for
Overcoming Obstacle, MERAL Portal.
Das, B. and D. Chakrabarty (2016). "Newton’s divided difference interpolation formula:
Representation of numerical data by a polynomial curve." International Journal of
Mathematics Trends and Technology-IJMTT 35.