Accuracy controlled iterative method for stops, and the final coefficients are used for approximating that

l coefficients are used for approximating that interval

efficient sigmoid function approximation y = bx + a (2)
✉
S.R. Chiluveru, M. Tripathy and B. Mohapatra
Start

The sigmoid activation function plays an essential role in implement-

ing neural networks in hardware. However, due to a high degree of temp = 0
non-linearity, the hardware implementation of the activation function
itself is a daunting job. In this Letter, the authors proposed an accuracy
controlled iterative algorithm for the piecewise linear approximation of xl = temp
xU = 8
the sigmoid function. The proposed algorithm approximates sigmoid
activation with an optimised number of linear functions for a given
error. The accuracy and hardware complexity of the resulting approxi-
f (x) = S(x), x [xl, xU]
mation function are compared with the state-of-the-art literature. y = a + bx

f (xi) – yi = (–1)i E
Introduction: The real-time implementation of the neural network (NN) i = 0,1,2.
E = ± max |f (x) – y |
got attention from the last decade [1], and it can be implemented as xl ≤ x ≤ xU
analogue or digital systems. However, the analogue implementation EM = maxi |Ei |

has got drawbacks like thermal drift, less computation accuracy, and EM = mini |Ei |

lack of re-programmability. Digital hardware implementation can over-

k = max |f (xi∗) – yi∗|
come these disadvantages [2]. Furthermore, the activation function plays i
xk = xk∗
a decisive role in the accuracy of NNs and its implementation [3].
The sigmoid activation function, being the function of exponential, is
non-linear. It is a challenging task to implement it in digital hardware No
EM == αEm?
[4]. However, the sigmoid activation functions has already been
approximated with various proposed methods like truncated series Yes

expansion [5], look-up tables (LUTs) [6], piecewise linear (PWL) No

EM ≤ ε ?
xU = xU – 0.05
approximation [7], and NN type approximation [8]. Amongst, the
PWL approximation scheme appears to be a good alternative that Yes
saves silicon area and shows short-latency time [9–11]. Research per-
No
formed in [3, 12] shows that the non-linear activation function xU == 8? temp = xU

implementation with higher accuracy improves the learning and gener-

Yes
alisation capabilities of NNs. However, implementations with higher
accuracy require more silicon area and decrease the network operation end
speed. Therefore, an optimised implementation of non-linear activation
function along with their derivatives is needed.
Stop
In this Letter, an accuracy selected PWL method is proposed, which
is used to implement the sigmoid function. The proposed method uses
Fig. 1 Flowchart for approximation of sigmoid activation function
an optimised number of first-order functions for the approximation of
sigmoid and its derivative. The coefficients of linear functions are
obtained by using the REMEZ optimisation algorithm recursively The proposed algorithm is shown in the dataflow model in Fig. 1.
with an allowable error constraint [13]. Afterwards, the hardware After finding the approximate function of the first interval, i.e.
implementation of the sigmoid is conducted, and it is validated with Ek ≤ 1, then the last maximum value of the first interval (xmax )
XOR-gate implementation with MLP on ZYNQ-7000 FPGA. becomes the initial value (xmin ) for the next interval, and the proposed
algorithm again searches for the next interval’s first-order function.
The process continues until it reaches the maximum limit point(8,0)
Iterative piecewise approximation model: In the proposed algorithm, the of the non-linear function. The derivative of sigmoid also found in
input domain of ( − 8, 8) is selected for approximating sigmoid function.
similar manner. The Table 1 values show the first-order coefficient
The sigmoid function has a range from 0 to 1, and it has got an inherent (i.e. a and b) for the approximation of sigmoid function. These values
symmetric property. It has an odd symmetry with respect to the point
are taken for the maximum error of 1 = 5 × 10−4 ; however, if this
(0,0.5) and hence obeys the expression f ( − x) = 1 − f (x). Therefore,
value changes, the corresponding intervals, and coefficient values
only a half part is sufficient to represent the entire function. The math- would also change.
ematical representation of sigmoid is shown in (1)

1 Table 1: First-order approximation coefficients for sigmoid

S(x) = (1) function
1 + e−x
Range a b Range a b
The proposed algorithm starts at the initial point, i.e. (0,0.5), and it [0–0.5] 0.5005 0.2449 [2.2–2.55] 0.7290 0.0781
uses a single value exchange REMEZ algorithm recursively. The seg- [0.5–0.8] 0.5103 0.2250 [2.55–2.95] 0.7834 0.0567
mentation of the non-linear function is performed automatically based [0.8–1.1] 0.5297 0.2009 [2.95–3.4] 0.8364 0.0388
on the required error limit of 1. Here, the 1 value is decided with the
[1.1–1.35] 0.5576 0.1755 [3.4–3.95] 0.8853 0.0244
acceptable error (precision) by NN.
[1.35–1.6] 0.5899 0.1516 [3.95–4.75] 0.9307 0.0129
The REMEZ algorithm is a kind of minimax algorithm, which pro-
[1.6–1.9] 0.6305 0.1262 [4.75–6.05] 0.9692 0.0048
cesses iteratively, and it operates on the Chebyshev min–max poly-
[1.9–2.2] 0.6781 0.1012 [6.05–8] 0.9916 0.0010
nomials [13]. The cost function, which is to be minimised in this
algorithm is the absolute maximum difference between the approxi-
mated polynomial and the actual function. For each interval, the non- Results and discussion: The analysis of sigmoid function is performed
linear function is approximated with a first-order polynomial, and it only for positive half part, and a total of 106 equidistance points are used
has a mathematical form like (2), where a is abscissa, and b is the in this range. The accuracy of the proposed implementations are
slope of the function. The REMEZ algorithm is made to converge to observed in terms of absolute average (Eavg ) and maximum (Emax )
a minimum of maximum error as shown in Fig. 1, where a is a constant errors, and they are defined as follows:
usually considered closer to 1, here, a = 1.05 [13]. Afterwards, if the 
−1 


f (xi ) − yi 
6
10
absolute maximum error is greater than the tolerable error (EM . 1), Eavg = (3)
then the range of sigmoid function updated (decreased), and the i=0
106
whole process is repeated. Otherwise, it means absolute maximum  
error less than or equal to tolerable error (EM ,= 1), then the process Emax = max f (xi ) − yi  (4)

ELECTRONICS LETTERS 3rd September 2020 Vol. 56 No. 18 pp. 914–916

Authorized licensed use limited to: Carleton University. Downloaded on September 13,2020 at 13:45:21 UTC from IEEE Xplore. Restrictions apply.
where f (xi ) is the discretised non-linear function value and yi is the Table 3: Precision of sigmoid with various bits
approximated function value for each point xi .
Precision Average error Maximum error
exact 2.64 × 10−4 4.99 × 10−4
Accuracy comparison: The comparative results of the proposed method
8 4.6 × 10−3 11.5 × 10−3
and published research are shown in Table 2. It is found that the pro-
10 0.7 × 10−3 2.3 × 10−3
posed method offers less average error compared to other methods.
Moreover, the maximum error is uniformly distributed for each interval 12 0.3 × 10−3 1.3 × 10−3
in the sigmoid function, as shown in Fig. 2. 14 2.6 × 10−4 6.8 × 10−4
16 2.4 × 10−4 5.2 × 10−4
1.0
True
0.8
Approx
distributed
% Error
memory
b-coefficient
x
0.6

0.4
X + Y

distributed
0.2 memory
a-coefficient

Fig. 3 Block diagram of hardware realisation of sigmoid function

–0.2
–8 –6 –4 –2 0 2 4 6 8

The sigmoid activation function is implemented on the PYNQ-Z1

Fig. 2 Sigmoid and its absolute maximum error board, which has a ZYNQ-7000 FPGA [18]. The hardware utilisation
of sigmoid and MLP implementations are shown in Table 4. Besides,
Table 2: Sigmoid function comparative results in terms of average the sigmoid and XOR-gate implemented entirely with combinational
error and maximum error logic, hence no clock is required. Finally, the sigmoid functionality is
No. of verified successfully on the PYNQ board.
Method Range ErrorAvg , (%) ErrorM , (%)
intervals
proposed [ − 8, 8] 0.0264 0.049 14 Table 4: Device utilisation summary for sigmoid and XOR-gate
Tommiska [2] [ − 8, 8] 0.17 0.39 —
Device utilisation summary
Myers and Hutchinson
[ − 8, 8] 2.47 4.90 7 Logic utilisation Used Available
[4]
PLAN [9] [ − 8, 8] 0.59 1.89 4 Sigmoid XOR-gate
Bajger and Omondi [10] [ − 8, 8] 0.1199 0.8616 16 slices 56 181 13,300
area method [11] [ − 8, 8] 0.1070 0.7880 16 LUTs 178 596 53,200
Tsai et al. [8] [ − 8, 8] 2.8 11.8 — BRAMs 0 0 144
Torki et al.[14] [ − 5, 5] 0.20 1.11 10 DSPs 1 5 220
Alippi and Storti-Gajani input/output 32 3 125
[ − 8, 8] 0.87 1.89 5
[15]
CRI q = 3 [16] [ − 8, 8] 0.85 2.06 24
Zhang et al. [17] [ − 4, 4] 0.77 2.16 — Conclusion: In this Letter, we have developed a sigmoid activation
function for efficient MLP implementation. The proposed approxi-
Hardware complexity comparison: The hardware complexity of mation uses an accuracy-based iterative model with the REMEZ algor-
the sigmoid function is compared in terms of the number of intervals ithm. The advantage of the proposed method is, it can work for any user
(i.e. first-order functions). Each interval requires one first-order function given tolerable error with the cost of hardware complexity. The func-
for sigmoid approximation; less number of first-order functions means tionality of the approximated sigmoid activation function verified suc-
less hardware complexity. Table 2 shows the number of intervals (or cessfully on FPGA.
first-order functions) required for sigmoid implementation; however,
no published literature gives an average error in the order of 10−4 . © The Institution of Engineering and Technology 2020
However, Zhang et al. [17] used 16-intervals and got an average error Submitted: 20 March 2020 E-first: 1 July 2020
of 5.7 × 10−4 , and each interval implemented with a second-order doi: 10.1049/el.2020.0854
equation. A neural network-based implementation is performed in [8] One or more of the Figures in this Letter are available in colour online.
and achieved an average error of 2.8 × 10−2 , and it requires large data
storage for implementation. In [2] the author uses a pure combinational S.R. Chiluveru, M. Tripathy and B. Mohapatra (Department of
implementation, and it has an average error of 2.8 × 10−2 . From all Electrical Engineering, Indian Institute of Technology Roorkee,
these results, it is observed that the proposed method provides a good Roorkee, India)
approximation for an average error of 2.6 × 10−4 using only ✉ E-mail: [email protected]
14-first-order functions.
References
Validation of sigmoid approximation: The validation of the sigmoid 1 Misra, J., and Saha, I.: ‘Artificial neural networks in hardware: a survey
function conducted by implementing the MLP model for XOR-gate of two decades of progress’, Neurocomputing, 2010, 74, (1–3),
implementation. The MLP model weights found offline, and these pp. 239–255
weights are converted to a fixed point format. The sigmoid function is 2 Tommiska, M.: ‘Efficient digital implementation of the sigmoid
developed with different fixed-point data widths like 8, 10, 12, 14, function for reprogrammable logic’, IEE Proc. Comput. Digit. Tech.,
and 16 bits. Table 3 shows the accuracy comparison of different fixed- 2003, 150, (6), pp. 403–411
point implementations. The 8-bit precision gives a significant deviation 3 Basterretxea, K., Tarela, J.M., del Campo, I., et al.: ‘An experimental
from exact value, while the 16-bit fixed-point representation gives study on nonlinear function computation for neural/fuzzy
hardware design’, IEEE Trans. Neural Netw., 2007, 18, (1),
nearly the exact values. Hence, the sigmoid function is used with a
pp. 266–283
16-bit fixed-point data type. 4 Myers, D., and Hutchinson, R.: ‘Efficient implementation of piecewise
The sigmoid function is realised with on-chip memory (distributed linear activation function for digital VLSI neural networks’, Electron.
memory). The Fig. 3 shows the block diagram for the realisation of Lett., 1989, 25, p. 1662
the sigmoid function, and it uses only first-order coefficients, which 5 Al-Nsour, M., and Abdel-Aty-Zohdy, H.S.: ‘Implementation of
are stored in distributed memory of FPGA. programmable digital sigmoid function circuit for neuro-computing’.

ELECTRONICS LETTERS 3rd September 2020 Vol. 56 No. 18 pp. 914–916

Authorized licensed use limited to: Carleton University. Downloaded on September 13,2020 at 13:45:21 UTC from IEEE Xplore. Restrictions apply.
 Proc. Midwest Symp. on Circuits and Systems (Cat. No. 98CB36268), 12 Basterretxea, K.: ‘Recursive sigmoidal neurons for adaptive accuracy
Notre Dame, IN, USA, 1998, pp. 571–574 neural network implementations’. Proc. NASA/ESA Conf. on
6 Piazza, F., Uncini, A., and Zenobi, M: ‘Neural networks with digital Adaptive Hardware and Systems (AHS), Erlangen, Germany, 2012,
LUT activation functions’. Proc. Int. Conf. on Neural Networks pp. 152–158
(IJCNN-93), Nagoya, Japan, 1993, vol. 2, pp. 1401–1404 13 Tawfik, S.A.: ‘Minimax approximation and REMEZ algorithm’.
7 Kim, Y., Kim, G., Hong, I., et al.: ‘A 4.9 MW neural network task sche- Available at http://wwwmathworkscom/matlabcentral/fileexchange/
duler for congestion-minimized network-on-chip in multi-core 8094-remez-algorithm 2005
systems’. Proc. IEEE Asian Solid-State Circuits Conf. (A-SSCC), 14 Torki, K., Faiedh, H., Souani, C., et al.: ‘Digital hardware implemen-
Kao Hsiung, Taiwan, 2014, pp. 213–216 tation of a neural system used for nonlinear adaptive prediction’, 2006
8 Tsai, C.H., Chih, Y.T., Wong, W.H., et al.: ‘A hardware-efficient 15 Alippi, C., and Storti-Gajani, G.: ‘Simple approximation of sigmoidal
sigmoid function with adjustable precision for a neural network functions: realistic design of digital neural networks capable of learn-
system’, IEEE Trans. Circuits Syst. II, Express Briefs, 2015, 62, (11), ing’. Proc. IEEE Int. Symp. on Circuits and Systems, Singapore,
pp. 1073–1077 Singapore, 1991, pp. 1505–1508
9 Amin, H., Curtis, K.M., and Hayes-Gill, B.R.: ‘Piecewise linear 16 Basterretxea, K., Tarela, J., and Del-Campo, I.: ‘Digital design of
approximation applied to nonlinear function of a neural network’, sigmoid approximator for artificial neural networks’, Electron. Lett.,
IEE Proc., Circuits Devices Syst., 1997, 144, (6), pp. 313–317 2002, 38, (1), pp. 35–37
10 Bajger, M., and Omondi, A.: ‘Low-error, high-speed approximation of 17 Zhang, M., Vassiliadis, S., and Delgado-Frias, J.G.: ‘Sigmoid genera-
the sigmoid function for large FPGA implementations’, J. Signal tors for neural computing using piecewise approximations’, IEEE
Process. Syst., 2008, 52, (2), pp. 137–151 Trans. Comput., 1996, 45, (9), pp. 1045–1049
11 Armato, A., Fanucci, L., Scilingo, E.P., et al.: ‘Low-error digital 18 Chen, C., Xia, J., Yang, W., et al.: ‘A PYNQ-compliant online platform
hardware implementation of artificial neuron activation functions for ZYNQ-based DNN developers’. Proc. ACM/SIGDA Int. Symp. on
and their derivative’, Microprocess. Microsyst., 2011, 35, (6), Field-Programmable Gate Arrays, Seaside, CA, USA, 2019,
pp. 557–567 pp. 185–185

ELECTRONICS LETTERS 3rd September 2020 Vol. 56 No. 18 pp. 914–916

Authorized licensed use limited to: Carleton University. Downloaded on September 13,2020 at 13:45:21 UTC from IEEE Xplore. Restrictions apply.