NN Lec - 04 - 05
NN Lec - 04 - 05
Lectures (4-5)
➢ The larger the input (more positive), the closer the output
value will be to 1.0, whereas the smaller the input (more
negative), the closer the output will be to 0.0.
➢ In Tanh, the larger the input (more positive), the closer the
output value will be to 1.0, whereas the smaller the input
(more negative), the closer the output will be to -1.0.
➢ Mathematically it can be represented as:
Advantages
⋮
(𝒏) (𝒏) (𝒏−𝟏) (𝒏)
𝒉𝒊 = 𝝓(𝒏) 𝒘𝒊𝒋 𝒉𝒊 + 𝒃𝒊
𝒋
𝝈 𝟐𝟎 × 𝟎 + 𝟐𝟎 × 𝟎 − 𝟏𝟎 = 𝝈 −𝟏𝟎 = 𝟎
(0,0) 𝝈 −𝟐𝟎 × 𝟎 − 𝟐𝟎 × 𝟎 + 𝟑𝟎 = 𝝈 𝟑𝟎 = 𝟏
𝝈 𝟐𝟎 × 𝟎 + 𝟐𝟎 × 𝟏 − 𝟑𝟎 = 𝝈 −𝟏𝟎 = 𝟎
𝝈 𝟐𝟎 × 𝟎 + 𝟐𝟎 × 𝟏 − 𝟏𝟎 = 𝝈 𝟏𝟎 = 𝟏
(0,1) 𝝈 −𝟐𝟎 × 𝟎 − 𝟐𝟎 × 𝟏 + 𝟑𝟎 = 𝝈 𝟏𝟎 = 𝟏
𝝈 𝟐𝟎 × 𝟏 + 𝟐𝟎 × 𝟏 − 𝟑𝟎 = 𝝈 𝟏𝟎 = 𝟏
𝝈 𝟐𝟎 × 𝟏 + 𝟐𝟎 × 𝟎 − 𝟏𝟎 = 𝝈 𝟏𝟎 = 𝟏
(1,0) 𝝈 −𝟐𝟎 × 𝟏 − 𝟐𝟎 × 𝟎 + 𝟑𝟎 = 𝝈 𝟏𝟎 = 𝟏
𝝈 𝟐𝟎 × 𝟏 + 𝟐𝟎 × 𝟏 − 𝟑𝟎 = 𝝈 𝟏𝟎 = 𝟏
𝝈 𝟐𝟎 × 𝟏 + 𝟐𝟎 × 𝟏 − 𝟏𝟎 = 𝝈 𝟑𝟎 = 𝟏
(0,0)
𝟏
𝝈 𝟐𝟎 × 𝟎 + 𝟐𝟎 × 𝟎 − 𝟏𝟎 = 𝝈 −𝟏𝟎 = = 𝟎. 𝟎𝟎𝟎𝟎𝟎𝟒𝟓 ≈ 𝟎
𝟏 + 𝒆−(−𝟏𝟎)
𝟏
𝝈 −𝟐𝟎 × 𝟎 − 𝟐𝟎 × 𝟎 + 𝟑𝟎 = 𝝈 𝟑𝟎 = ≈𝟏
𝟏 + 𝒆−(𝟑𝟎)
𝟏
𝝈 𝟐𝟎 × 𝟎 + 𝟐𝟎 × 𝟏 − 𝟑𝟎 = 𝝈 −𝟏𝟎 = = 𝟎. 𝟎𝟎𝟎𝟎𝟎𝟒𝟓 ≈ 𝟎
𝟏 + 𝒆−(−𝟏𝟎)
(0,1)
𝟏
𝝈 𝟐𝟎 × 𝟎 + 𝟐𝟎 × 𝟏 − 𝟏𝟎 = 𝝈 𝟏𝟎 = = 𝟎. 𝟗𝟗𝟗𝟗𝟓𝟓 ≈ 𝟏
𝟏 + 𝒆−(𝟏𝟎)
𝟏
𝝈 −𝟐𝟎 × 𝟎 − 𝟐𝟎 × 𝟏 + 𝟑𝟎 = 𝝈 𝟏𝟎 = = 𝟎. 𝟗𝟗𝟗𝟗𝟓𝟓 ≈ 𝟏
𝟏 + 𝒆−(𝟏𝟎)
𝟏
𝝈 𝟐𝟎 × 𝟏 + 𝟐𝟎 × 𝟏 − 𝟑𝟎 = 𝝈 𝟏𝟎 = = 𝟎. 𝟗𝟗𝟗𝟗𝟓𝟓 ≈ 𝟏
𝟏 + 𝒆−(𝟏𝟎)
(1,0)
𝟏
𝝈 𝟐𝟎 × 𝟏 + 𝟐𝟎 × 𝟎 − 𝟏𝟎 = 𝝈 𝟏𝟎 = = 𝟎. 𝟗𝟗𝟗𝟗𝟓𝟓 ≈ 𝟏
𝟏 + 𝒆−(𝟏𝟎)
𝟏
𝝈 −𝟐𝟎 × 𝟏 − 𝟐𝟎 × 𝟎 + 𝟑𝟎 = 𝝈 𝟏𝟎 = = 𝟎. 𝟗𝟗𝟗𝟗𝟓𝟓 ≈ 𝟏
𝟏 + 𝒆−(𝟏𝟎)
𝟏
𝝈 𝟐𝟎 × 𝟏 + 𝟐𝟎 × 𝟏 − 𝟑𝟎 = 𝝈 𝟏𝟎 = = 𝟎. 𝟗𝟗𝟗𝟗𝟓𝟓 ≈ 𝟏
𝟏 + 𝒆−(𝟏𝟎)
(1,1)
𝟏
𝝈 𝟐𝟎 × 𝟏 + 𝟐𝟎 × 𝟏 − 𝟏𝟎 = 𝝈 𝟑𝟎 = =𝟏
𝟏 + 𝒆−(𝟑𝟎)
𝟏
𝝈 −𝟐𝟎 × 𝟏 − 𝟐𝟎 × 𝟏 + 𝟑𝟎 = 𝝈 −𝟏𝟎 = = 𝟎. 𝟎𝟎𝟎𝟎𝟒𝟓 ≈ 𝟎
𝟏 + 𝒆−(−𝟏𝟎)
𝟏
𝝈 𝟐𝟎 × 𝟏 + 𝟐𝟎 × 𝟎 − 𝟑𝟎 = 𝝈 −𝟏𝟎 = = 𝟎. 𝟎𝟎𝟎𝟎𝟒𝟓 ≈ 𝟎
𝟏 + 𝒆−(−𝟏𝟎)
𝒘𝟏 = 𝟎. 𝟒 𝒘𝟓 = 𝟎. 𝟔
𝟎. 𝟒 𝒙𝟏 𝒉𝟏𝟏 𝒉𝟐𝟏
𝒚𝟏
𝒚𝟐
𝟎. 𝟒 𝒙𝟐 𝒉𝟏𝟐 𝒉𝟐𝟐
𝒘𝟒 = 𝟎. 𝟒 𝒘𝟖 = 𝟎. 𝟔
RELU TANH Softmax
𝒚𝟏
𝒚𝟐
𝟎. 𝟒 𝒙𝟐 𝒉𝟏𝟐 𝒉𝟐𝟐
𝒘𝟒 = 𝟎. 𝟒 𝒘𝟖 = 𝟎. 𝟔
RELU TANH Softmax
𝒉𝟏𝟏 = 𝝈 𝒘𝟏 × 𝒙𝟏 + 𝒘𝟑 × 𝒙𝟐 = 𝝈 𝟎. 𝟒 × 𝟎. 𝟒 + 𝟎. 𝟔 × 𝟎. 𝟒
= 𝝈 𝟎. 𝟒 = 𝐦𝐚𝐱 𝟎, 𝟎. 𝟒 = 𝟎. 𝟒
𝒉𝟏𝟐 = 𝝈 𝒘𝟐 × 𝒙𝟏 + 𝒘𝟒 × 𝒙𝟐 = 𝝈 𝟎. 𝟔 × 𝟎. 𝟒 + 𝟎. 𝟒 × 𝟎. 𝟒
= 𝝈 𝟎. 𝟒 = 𝐦𝐚𝐱 𝟎, 𝟎. 𝟒 = 𝟎. 𝟒
𝒚𝟏
𝒚𝟐
𝟎. 𝟒 𝒙𝟐 𝒉𝟏𝟐 𝒉𝟐𝟐
𝒘𝟒 = 𝟎. 𝟒 𝒘𝟖 = 𝟎. 𝟔
RELU TANH Softmax
𝒚𝟏
𝒚𝟐
𝟎. 𝟒 𝒙𝟐 𝒉𝟏𝟐 𝒉𝟐𝟐
𝒘𝟒 = 𝟎. 𝟒 𝒘𝟖 = 𝟎. 𝟔
RELU TANH Softmax
𝒆𝟎.𝟕𝟐
𝒚𝟏 = 𝐒𝐨𝐟𝐭𝐦𝐚𝐱 𝟎. 𝟕𝟐 = = 𝟎. 𝟓𝟒
𝒆𝟎.𝟕𝟐 + 𝒆𝟎.𝟓𝟕𝟔
𝒆𝟎.𝟓𝟕𝟔
𝒚𝟐 = 𝐒𝐨𝐟𝐭𝐦𝐚𝐱 𝟎. 𝟓𝟕𝟔 = 𝟎.𝟕𝟐 = 𝟎. 𝟒𝟔
𝒆 + 𝒆𝟎.𝟓𝟕𝟔
1: [0, 1, 0, 0, 0, 0, 0, 0, 0, 0] 7: [0, 0, 0, 0, 0, 0, 0, 1, 0, 0]
2: [0, 0, 1, 0, 0, 0, 0, 0, 0, 0] 8: [0, 0, 0, 0, 0, 0, 0, 0, 1, 0]
3: [0, 0, 0, 1, 0, 0, 0, 0, 0, 0] 9: [0, 0, 0, 0, 0, 0, 0, 0, 0, 1]
4: [0, 0, 0, 0, 1, 0, 0, 0, 0, 0]
5: [0, 0, 0, 0, 0, 1, 0, 0, 0, 0]
Line 2 loads the MNIST dataset from disk. If you have never run
this function before, then the MNIST dataset will be downloaded
and stored locally to your machine. Once the dataset has been
downloaded, it is cached to your machine and will not have to be
downloaded again.