Scribd
Upload
6 views20 pages

Lecture 2-Convolutional Networks

The document discusses convolutional networks (CNNs), which are specialized neural networks designed for processing grid-like data using convolution operations instead of general matrix multiplication. It explains the convolutional operation, including the use of weighted averages to smooth data and the distinction between input, kernel, and feature map in the context of machine learning. Additionally, it highlights the commutative property of convolution and its practical implementation in neural network libraries, often using cross-correlation instead.

Uploaded by

ashish.k
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
6 views20 pages

Lecture 2-Convolutional Networks

The document discusses convolutional networks (CNNs), which are specialized neural networks designed for processing grid-like data using convolution operations instead of general matrix multiplication. It explains the convolutional operation, including the use of weighted averages to smooth data and the distinction between input, kernel, and feature map in the context of machine learning. Additionally, it highlights the commutative property of convolution and its practical implementation in neural network libraries, often using cross-correlation instead.

Uploaded by

ashish.k
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 20

Convolutional Networks

By

Atanendu Sekhar Mandal


Morphology of Neurons
Brain Circuitry & Learning
Deep Networks
• Modern deep learning provides a very powerful framework
for supervised learning

• By adding more layers and more units within a layer, a deep


network can represent functions of increasing complexity

• Most tasks that consist of mapping an input vector to an


output vector, simple tasks for an individual, can be
accomplished via deep learning, given sufficiently large
models and sufficiently large datasets of labeled training
examples

• Other tasks, that can not be described as associating one


vector to another, difficult tasks for humans, remain
beyond the scope of deep learning for now
Deep Networks
• The core parametric function approximation
technology is behind nearly all modern practical
applications of deep learning

• The feedforward deep network model is used to


represent these functions

• Advanced techniques for regularization and


optimization of such models are needed
Deep Networks
• The convolutional network can be used for
scaling to large images

• The recurrent neural network can be used for


processing temporal sequences
CNN - Introduction
• Convolutional networks (LeCun, 1989), also known as
neural networks or CNNs, are a specialized kind of
neural network for processing data that has a known,
grid-like topology

• The name “convolutional neural network” indicates


that the network employs a mathematical operation
called convolution

• Convolutional networks are simply neural networks


that use convolution in place of general matrix
multiplication in at least one of their layers.
The Convolutional Operation
• Suppose we are tracking the location of a
spaceship with a laser sensor

• Our laser sensor provides a single output x(t), the


position of the spaceship at time t

• Both x and t are real-valued, i.e., we can get a


different reading from the laser sensor at any
instant in time
The Convolutional Operation
• Now suppose that our laser sensor is somewhat noisy

• To obtain a less noisy estimate of the spaceship’s


position, we would like to average together several
measurements

• Of course, more recent measurements are more


relevant, so we will want this to be a weighted average
that gives more weight to recent measurements

• We can do this with a weighting function w(a), where a


is the age of a measurement
The Convolutional Operation
• If we apply such a weighted average operation
at every moment, we obtain a new function
providing a smoothed estimate of the position
s of the spaceship:

s ( t ) =  x ( a )w ( t − a ) da (1)

• This operation is called convolution


The Convolutional Operation
• The convolution operation is typically denoted
with an asterisk:
s ( t ) = ( x * w )( t ) ( 2)

• In our example, w needs to be a valid probability


density function

• Also, w needs to be 0 for all negative arguments


The Convolutional Operation
• In convolutional network terminology, the first
argument (in this example, the function x) to the
convolution is often referred to as the input

• The second argument (in this example, the


function w) as the kernel

• The output is sometimes referred to as the


feature map
The Convolutional Operation
• The idea of a laser sensor that can provide
measurements at every instant in time is not
realistic

• Our sensor will provide data at regular intervals.


It might be more realistic to assume that our
laser provides a measurement once per second

• The time index t can then take on only integer


values
The Convolutional Operation
• If we now assume that x and w are defined only
on integer t, we can define the discrete
convolution:

s ( t ) = ( x * w )( t ) =  x ( a )w (t − a )
a =−
( 3)

• In machine learning applications, the input is


usually a multidimensional array of data and the
kernel is usually a multidimensional array of
parameters that are adapted by the learning
algorithm
The Convolutional Operation
• We will refer to these multidimensional arrays as
tensors

• Because each element of the input and kernel must be


explicitly stored separately, we usually assume that
these functions are zero everywhere except over the
finite set of points for which we store the values

• This means that in practice we can implement the


infinite summation as a summation over a finite
number of array elements
The Convolutional Operation
• Finally, we often use convolutions over more
than one axis at a time

• For example, if we use a two-dimensional


image I as our input, we probably also want to
use a two-dimensional kernel K:
S ( i, j ) = ( I * K )( i, j ) =  I ( m, n )K ( i − m, j − n ) ( 4)
m n
The Convolutional Operation
• Convolution is commutative, meaning we can
equivalently write:
S ( i, j ) = ( K * I )( i, j ) =  I ( i − m, j − n )K ( m, n ) (5)
m n

• Usually this formula is more straightforward to


implement in a machine learning library

• Because there is less variation in the range of


valid values of m and n.
The Convolutional Operation
• The commutative property of convolution arises
because we have flipped the kernel relative to the
input, in the sense that as m increases, the index into
the input increases, but the index into the kernel
decreases

• The only reason to flip the kernel is to obtain the


commutative property

• The commutative property it is not usually an


important property of a neural network
implementation.
The Convolutional Operation
• Instead, many neural network libraries
implement a related function called the cross-
correlation, which is the same as convolution
but without flipping the kernel:

S ( i, j ) = ( I * K )( i, j ) =  I ( i + m, j + n )K ( m, n ) (6)
m n

• Many machine learning libraries implement


cross-correlation but call it convolution
The Convolutional Operation
• Fig. 1 for an example of convolution (without
kernel flipping) applied to a 2-D tensor

You might also like