Scribd
Upload
85 views11 pages

Convolution and Pooling As An Infinitely Strong Prior

The document discusses the concept of prior probability distributions in neural networks, specifically focusing on convolution and pooling as infinitely strong priors that impose strict constraints on model parameters. It explains how these priors guide the learning process by enforcing local interactions and translation invariance, but also highlights potential issues such as underfitting when precise spatial information is necessary. The text suggests that a balanced approach, using pooling selectively, can help maintain both robust and detailed features in convolutional neural networks.

Uploaded by

devanand272003
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
85 views11 pages

Convolution and Pooling As An Infinitely Strong Prior

The document discusses the concept of prior probability distributions in neural networks, specifically focusing on convolution and pooling as infinitely strong priors that impose strict constraints on model parameters. It explains how these priors guide the learning process by enforcing local interactions and translation invariance, but also highlights potential issues such as underfitting when precise spatial information is necessary. The text suggests that a balanced approach, using pooling selectively, can help maintain both robust and detailed features in convolutional neural networks.

Uploaded by

devanand272003
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 11

Convolution and Pooling as an

infinitely strong prior

Mr. Sivadasan E T
Associate Professor
Vidya Academy of Science and Technology, Thrissur
Prior probability distribution
in neural networks

A prior is an assumption a model makes before


learning from data.
In neural networks, priors guide what patterns the
model should focus on.
Infinitely strong prior

An infinitely strong prior places zero probability on


some parameters and says that these parameter
values are completely forbidden, regardless of
how much support the data gives to those values.
Convolution as a Strong Prior

We can imagine a convolutional net as being


similar to a fully connected net, but with an
infinitely strong prior over its weights.

This infinitely strong prior says that the weights for


one hidden unit must be identical to the weights
of its neighbor, but shifted in space.
Convolution as a Strong Prior

The prior also says that the weights must be zero,


except for in the small, spatially contiguous
receptive field assigned to that hidden unit.

Overall, we can think of the use of convolution as


introducing an infinitely strong prior probability
distribution over the parameters of a layer.
Convolution as a Strong Prior

This prior says that the function the layer should


learn contains only local interactions and is
equivariant to translation.
Pooling as a Strong Prior

Likewise, the use of pooling is an infinitely strong


prior that each unit should be invariant to small
translations.
Convolution and Pooling as an
infinitely strong prior
One key insight is that convolution and pooling
can cause underfitting.
Like any prior, convolution and pooling are only
useful when the assumptions made by the prior
are reasonably accurate.
If a task relies on preserving precise spatial
information, then using pooling on all features can
increase the training error.
Convolution and Pooling as an
infinitely strong prior

Some CNNs use pooling on some channels but


not others to get both robust features (from
pooled channels) and detailed features (from
non-pooled channels), avoiding underfitting
when translation invariance doesn't work well.
Convolution and Pooling as an
infinitely strong prior

When a task involves incorporating information


from very distant locations in the input, then the
prior imposed by convolution may be
inappropriate.
Thank You!

You might also like