OLMP Lab6
OLMP Lab6
OLMP Lab6
Lab Courses
Lab 6: OLMP
OLMP
OLMP
LMP
ANN
ANN
• The study of artificial neural networks was inspired by attempts to simulate biological neural
systems (e.g. human brain).
• Basic structural and functional unit: nerve cells called neurons
• Work Mechanism
• Different neurons are linked together via axons (轴突) and dendrite (树突)
• When one neuron is excited after stimulation, it sends chemicals to the connected neurons,
thereby changing the potential (电位) within these neurons.
• If the potential (电位) of a neuron exceeds a “threshold”, then it is activated and send
chemicals to other neurons.
ANN
• A neuron is connected to the axons (轴突) of other neurons via dendrites (树突), which are
extensions from the cell body of the neuron.
• The contact point between a dendrite (树突) and an axon (轴突) is called a synapse (突触).
• The human brain learns by changing the strength of the synaptic connection between
neurons upon repeated stimulation by the same impulse.
Artificial Neuron Mathematical Model
(*) * (3) 3 4
𝑧7 = 𝑓(a7 ) 𝑧7 = 𝑓(a7 ) 𝑦= = 𝑓 a=
9
(*) (*)
a7 = 8 𝑤$" 𝑥" − 𝜃$
")*
<
(3) (3)
a; = 8 𝑤;$ 𝑧$ − 𝜃;
$)*
>
Superscript of w:layer index (4) (4)
a= = 8 𝑤=; 𝑧; − 𝜃=
;)*
Training of NN
x1 x2 x3 l1 l2
1.0 0.1 0.3 1 0 𝐸𝑟𝑟𝑜𝑟 𝐹𝑢𝑛𝑐𝑡𝑖𝑜𝑛 𝑊, 𝜃
1
0.1 1.5 1.2 1 0 = [ 𝑜1 𝑊, 𝜃 − 𝑦1 3
2
1.1 1.1 2.0 0 1 + 𝑜2 𝑊, 𝜃 − 𝑦2 3 ]
0.2 0.2 0.3 0 1
Calculate the gradient for
𝑊, 𝜃 , then tune them
Deep Neural Networks
ZIP
IPhone 8
=289MB 2GB RAM =104MB
=333MB =125MB
DNN Compression
Where:
• 𝑛= is the number of neurons in layer 𝑙, 𝑛=U* is similar.
=
• 𝑊",$ denotes the connection weight between the 𝑖 WX neuron in layer 𝑙 and the 𝑗WX
neuron in layer 𝑙 + 1.
=
• 𝑊",$ = 0 indicates the corresponding connection does not exist.
Layer 𝑙 + 1 𝑗
𝑛=U* neurons
=
𝑊",$
Layer 𝑙 𝑖
𝑛= neurons
MP and OLMP
Magnitude-based pruning (MP) [LeCun et al., 1989] : Given a network 𝑊 and a threshold 𝜀,
Magnitude-based pruning indicates:
𝑀𝑃 𝑊, 𝜀 = 𝑤 𝑤 ≥ 𝜀, 𝑤 ∈ 𝑊 .
This method prunes the connections whose absolute connection weights are lower than 𝜀.
Layer-wise magnitude-based pruning (LMP) [Guo et al., 2016; Han et al., 2015] : Instead of apply MP on
the whole network, LMP applies one each layer separately:
𝐿𝑀𝑃 𝑊, {𝜀* , 𝜀3 , … , 𝜀` } = ⋃`=)* 𝑀𝑃 𝑊, 𝑙, 𝜀= , (2)
= =
where 𝑀𝑃 𝑊, 𝑙, 𝜀 = 𝑊",$ 𝑊",$ ≥ 𝜀, 1 ≤ 𝑖 ≤ 𝑛= , 1 ≤ 𝑗 ≤ 𝑛=U* .
Derivative-free optimization methods [Goldberg, 1989; Brochu et al., 2010; Qian et al., 2015; Yu et al., 2016] : do not require
the problem to be either continuous or differentiable. In our paper, we use negatively
correlated search (NCS) [Tang et al., 2016] to solve Eq. (3).
Negatively Correlated Search [Tang et al., 2016] :It uses negative correlations to increase diversities
among solutions and to encourage them to search different areas of the solution space.
= =
Eq. (1): 𝑊 = 𝑊",$ 𝑊",$ ≠ 0,1 ≤ 𝑙 ≤ 𝐿, 1 ≤ 𝑖 ≤ 𝑛= , 1 ≤ 𝑗 ≤ 𝑛=U* .