A Seminar Report on

Smart Cameras As Embedded System

For the Degree of
Bachelor of Technology
In
Electronics and Telecommunication Engineering
Submitted by
By Komal Dattarao Nirwal
(ID-19004020)
Supervised by
Prof. R.R.Harne

Department of Electronics and telecommunication Engineering

Government College of Engineering, Amravati
(An Autonomous Institute of Government of Maharashtra)
Maharashtra State, India
2022-2023
 GOVERNMENT COLLEGE OF ENGINEERING AMRAVATI
(An Autonomous Institute of Government of Maharashtra)
Department of Electronics and Telecommunication Engineering

CERTIFICATE

This is to certify that Komal Nirwal student of Fourth Year B. Tech in

Electronics and Telecommunication Engineering, has submitted this
seminar report for the “Smart Cameras as Embedded System”, which is
being submitted Herewith for the award of the Degree of Bachelor of
Technology in Electronics and Telecommunication Engineering of
Government College of Engineering, Amravati, is The result of the work
completed under my supervision and guidance within the institute And
the same has not been submitted elsewhere for the award of any
degree.

Guide Head OF Department

Prof.R.R.Harne P.R.Deshmukh
 DECLARATION

• I hereby declare that this seminar report entitled as Smart Cameras in

Embedded System is based on my own work carried out during the
course of our study under the supervision of Prof R.R.Harne. I assert the
statements made and conclusions drawn are an outcome of my
research work. I further certify that
• The work contained in the report is original and has been done by me
under the general supervision of my supervisor.
• The work has not been submitted to any other Institution for any
other degree, diploma certificate in this university or any other
University of India or abroad
• Whenever we have used materials (data, theoretical analysis, and
text) from other sources, we have given due credit to them in the text
of the report and giving their details in the references.

Place: Amravati Komal Nirwal

Date: 19004020
 ACKNOWLEDGEMENT

In this part of report, I like to express my deep sense of gratitude and

sincere thanks to All those illumination because of whom, this work has
become successful. I am highly indebted to Prof. R.R.Harne for their
guidance and constant supervision as well as for providing necessary
information regarding the project and also for their support in
completing the project. I would like to express my special gratitude and
thanks to all M. Tech students for giving me such attention and time. I
also acknowledge and thank to principal sir and all the staff members of
Electronics And Telecommunication Department, Library and Workshop
Department who has directly and, indirectly helped.

Place:Amravati Komal Nirwal

Date:28/11/2022 (ID-19004020)
 ABSTRACT

A smart camera performs real-time analysis to recognize scenic

elements. Smart cameras are useful in a variety of scenarios:
surveillance, medicine.

We have built a real-time system for recognizing gestures. Our smart

camera uses novel algorithms to recognize gestures based on low-level
analysis of body parts as well as hidden Markov models for the moves
that comprise the gestures.

These algorithms run on a Trimedia processor. Our system can

recognize gestures at the rate of 20 frames/second. The camera can
also fuse the results of multiple cameras
 CONTENTS
1. Introduction 3

1.1 Introduction 4
2. A Smart Camera 6

2.1 Overview of smart camera 7

2.2 Smart camera V/s standard smart vision system 8

2.3 2.3.1 PC based vision system advantages 9

2.3.2 Smart camera's advantages 9

2.4 Smart camera architecture 10

2.4.1 Image sensor basics 11
2.4.2 Analog to digital conversion electronics 12
2.4.3 Image processing electronics 12

2.5 Image processing algorithms 13

2.5.1 Static grey scale thresholding 14
2.5.2 Embedded image processing algorithm 15
2.6 Smart camera system 16

3. Embedded systems 17

3.1 Introduction to embedded systems 18

3.2 Examples of embedded systems 19

3.3 Characteristics of embedded systems 21

4. Smart cameras as embedded systems 22

Page no.1
 4.1 Overview of SCES 23
4.2 Detection and recognition algorithms 23
4.3 Requirements 28
4.4 Components 28
4.5 Testbed architecture 28
4.6 Experiments and optimizations 29
4.7 Algorithmic changes 30
4.8 Control-to-data transmission 31

5. Applications of smart cameras 32

5.1 Fields of application 33

5.2 Application of smart camera networks 34
5.2.1 Visual hull reconstruction 34

6. Conclusion & some future trends 36

6.1 Optimization results & conclusion 37
6.2 Looking Ahead 38
6.2.1 Future applications 38

Referance 41

Page no.2
1.INTRODUCTION

SMART CAMERAS In EMBEDDED SYSTEMS-

Man is striving to make the machines do tasks that

humans can do or try to do through embedded distributed
intelligence.

Page no.3
1.1 Introduction

A smart camera performs real-time analysis to recognize scenic elements. Smart

cameras are useful in a variety of scenarios: surveillance, medicine, etc. We have
built a real-time system for recognizing gestures. Our smart camera uses novel
algorithms to recognize gestures based on low-level analysis of body parts as well
as hidden Markov models for the moves that comprise the gestures. These
algorithms run on a Trimedia processor. Our system can recognize gestures at the
rate of 20 frames/second. The camera can also fuse the results of multiple cameras.

A "smart camera" is basically a video camera coupled to a computer vision system

in a tiny package. This communication begins stating the main differences between
smart cameras and standard smart vision systems. Smart camera architecture is
described whereby a combination of an on-board microprocessor and PLD's allow
for the embedding of image processing algorithms in the camera. A static
thresholding algorithm is presented which demonstrates the ability to track non-
uniformity in the inspection target. A multi camera inspection system application is
presented where a maximum of twenty smart cameras may be networked together
to a single host computer. Finally, a prediction is made of technological and
applicational future evolution on smart cameras.

The smart camera - a whole vision system contained in one neat housing - can be
used anywhere, in any industry where image processing can be applied. Companies
no longer need a cabinet in which to keep all their computing equipment: the
computer is housed within the smart camera. In the pharmaceutical industry and in
clean rooms when not even dust is allowed - this can be a big advantage. A single
square meter of space can be comparatively very expensive-if there is no need for a
component rack or cabinet, simply a smart camera, and then this could save a lot of
money. In particular, there would not be the usual cabling involved for other vision
systems, and set-up is simple. Later in this communication are stated some
advantages of using smart cameras or PC-based systems in vision applications.

In usual vision systems scenarios, only a small fraction of a picture frame will be
the region of interest (ROI). In fact, often no visual image of the ROI is even
required. The object of a vision system, after all, is to make a decision: "Is there a
blob"? "Where is the blob"? "Is this a

Page no.4
What if all that pixel pre-processing and decision-making could be done within the
camera? If all the processing were done inside the camera, the blob analysis of a
gigabit image might result in only a few hundred bytes of data which need to be
sent somewhere. Such compact packets of data could be easily transmitted directly
to a machine control without even passing through a PC. Information should be
processed where the information occurs - Le. the brain should be behind

the eyes!

The answer to the problem stated earlier is a stand-alone, smart camera. To

illustrates this, a smart camera's embedded system architecture is shown along with
an example of a hardware

embedded image processing algorithm: "Static Gray Scale Thresholding". Many

global companies including Glaxo and Ciba-Geigy, and motor companies such as
Ford and

Volkswagen are users of smart cameras. A small example of a vision system for
web inspection is shown where twenty smart cameras are connected together.

Many changes are yet to come concerning smart cameras as in technology as well
as in there future applications.

Page no.5
2.A Smart Camera

"smart camera" for an image capture system with embedded computing

can extract information from images without need for an external
processing unit, and interface devices used to make results available to
other devices. - RETICON (first smart camera users)

Page no.6
2.1 OVERVIEW OF SMART CAMERA

A smart camera is an integrated machine vision system which, in addition to image

capture circuitry, includes a processor, which can extract information from images
without need for an external processing unit, and interface devices used to make
results available to other devices.

A smart camera or "intelligent camera" is a self-contained, standalone vision

system with built-in
image sensor in the housing of an industrial video camera It contains all necessary
communication interfaces, eg. Ethernet, as well as industry-proof 24V I/O lines for
connection to a PLC, actuators, relays or pneumatic valves. It is not necessarily
larger than an industrial or surveillance camera.

This architecture has the advantage of a more compact volume compared to PC-
based vision systems and often achieves lower cost, at the expense of a somewhat
simpler (or missing altogether) user interface.

Although often used for simpler applications, modern smart cameras can rival PCs
in terms of

processing power and functionalities. Smart cameras have been marketed since the
mid 80s, but only in recent years have they reached widespread use, once
technology allowed their size to be reduced while their processing power has
reached several thousand MIPS (devices with IGHZ processors and up to
8000MIPS are available as of end of 2006).

Having a dedicated processor in each unit, smart cameras are especially suited for
applications where several cameras must operate independently and often
asynchronously, or when distributed vision is required (multiple inspection or
surveillance points along a production line or within an assembly machine).

Fig 2.1 Early smart camera (ca. 1985, in red) with an 8MHZ Z80 compared to a
modern device featuring Texas Instruments' C64 @1GHz

Page no.7
A smart camera usually consists of several (but not necessarily all) of the following
components:
• Image sensor (matrix or linear, CCD- or CMOS)
• Image digitization circuitry
• Image memory
• processor (often a DSP or suitably powerful processor)
• program- and data memory (RAM, nonvolatile FLASH) • Communication
interface (RS232, Ethernet)
• IO lines (often opt isolated)
• Lens holder or built in lens (usually C, CS or M-mount)
• Built in illumination device (usually LED) • Purpose developed real-time
operating system (For example VCRT)

A video output (e.g. VGA or SVGA) may be an option for a Smart Camera.

2.2 Smart Cameras vs. Standard Smart Vision Systems

The question often comes up as to what is the most appropriate approach to take in
implementing a vision system - using a smart camera or using some sort of PC-
based approach. There is no question that as the microprocessors, DSPs and
FPGAS are getting faster and, therefore, more capable, smart cameras are getting
smarter. Therefore, they are a challenge to more "traditional" approaches to vision
systems. Significantly, however, "traditional" approaches are also taking advantage
of the advances and so, too, are faster and smarter.

Traditional approaches usually mean a PC-based implementation. This could be

either using a camera with the capability to interface directly to the PC (IEEE
1394/Fire wire, Camera Link. LVDS, USB, etc.), or a system based on a frame
grabber or other intelligent image processing board or vision engine that plugs into
the PC. In this latter case, more conventional analog cameras are used as the input
device.
A smart camera, on the other hand, is a self-contained unit. It includes the imager
as well as the "intelligence" and related I/O capabilities. Because this format
resembles the format of many intelligent sensors, these products are often referred
to as "vision sensors." However, a vision sensor often has a limited and fixed
performance envelope, while a smart camera has more flexibility or tools,
inherently capable of being programmed to handle many imaging algorithms
Page no.8
and application functions. A PC-based vision system is generally recognized as
having the greatest flexibility and, therefore, capable of handling a wider range of
applications.

2.3.1 PC-Based Vision Systems Advantages

The PC-based vision systems advantages include:

1) Flexibility-

The PC offers greater flexibility in the number of options that can be selected.
For example one can use a line scan versus an area scan camera with the PC. One
can use third party software packages with the PC approach (smart cameras tend to
be single source software).

2) Power-

PC's tend to offer greater power and speed due in large part to the speed of the Intel
processors used internally. This power in turn means that PC's are used to handle
the "tougher" applications in vision systems.

2.3.2 Smart Cameras Advantages

The smart cameras advantages include:

1) Cost-

Smart cameras are generally less expensive to purchase and set up than the PC
based solution, since they include the camera, lenses, lighting (sometimes), cabling
and processing.

2) Simplicity-

Software tools available with smart cameras are of the point-and-click variety and
are easier to use than those available on PC's. Algorithms come pre-packaged and
do not need to be developed, thus making the smart camera quicker to setup and
use.

3) Integration –
Page no.9
Given their unified packaging, smart cameras are easier to integrate into the
manufacturing environment.

4) Reliability -

With fewer moving components (fans, hard drives) and lower temperatures, smart
cameras are more reliable than PC's.

In general the performance of the smart camera will continue to increase. This will
mean that the

smart camera will be used for more difficult applications, slowly displacing the PC
approach.

Fig 2.2. Standard Smart Vision System

Page no.10
Fig 2.3. Smart Camera System.

2.4 Smart Camera Architecture

The smart camera presented in this communication reduces the amount of data
generated to the "data of interest by making use of embedded image processing
algorithms. The data of interest might be, for example, defective areas of the
product being inspected. Multiple cameras can route their data to a single frame
grabber and computer due to the reduction of data stream, thus dramatically
reducing system cost and increasing inspection bandwidth capability. This smart
camera also makes use of an on-board microprocessor for communication with the
inspection systems' host computer and for internal control functions:

The following block diagram illustrates the camera architecture

Fig 2.4. Smart Camera Architecture Block Diagram

A detailed explanation of the camera architecture follows, starting with the image
sensor.

Page no.11
2.4.1 Image Sensor Basics

In this smart camera, a CCD (Charge Coupled Device) image sensor converts
photons (light) into electrons (charge). When photons hit an image sensor, the
sensor accumulates electrons. This is called charge integration. The brighter your
light source, the more photons available for the sensor to integrate, and the smaller
the amount of time required to collect a given amount of light energy.

Finally, the sensor transfers its aggregate charge to readout registers, which feed
each pixel's charge from the image sensor into an output node that converts the
charges into voltages. After this transfer and conversion, the voltages are amplified
to become the camera's analog output

2.4.2 Analog to Digital Conversion Electronics

The analog output of the CCD is converted to a digital output for further
processing. The camera presented here sub-divides the CCD analog output into
eight channels of 256 pixel elements each. Analog to digital conversion is
performed at a 20 MHz data rate for each channel thus yielding an effective camera
data rate of 160 MHz. The digital data is then passed along to the image processing
electronics for processing and analysis.

2.4.3 Image Processing Electronics

Image processing is performed by embedded algorithms on a per channel basis.

The following block diagram illustrates the basic processing architecture for each
channel.

Page no.12
Fig 2.5. Image Processing Architecture Block Diagram

The processing algorithm is embedded in the processing PLD. The microprocessor

has a bidirectional path being able to randomly access the algorithm parameters, as
well as program a new algorithm into the PLD as required by the user. Raw pixel
data and associated timing and control signals are also connected to input pins into
the processing PLD.. For storage and subsequent readout, algorithm processed data
is output along with a write control signal to FIFO memory. 8:1 multiplexing of the
data is achieved by using FIFO memory. Readout control is accomplished by the
microprocessor/FIFO readout control card whose architecture is shown in the
figure 2.6.

Fig 2.6. Microprocessor/FIFO Readout Control Circuit Board Block Diagram

Page no.13
The Microprocessor/FIFO readout control circuit board acts as the master
controller for the smart camera. FLASH memory is used to store microprocessor
code and PLD algorithm code. In system programmability is achieved because all
PLD devices in the image processing section of the camera are SRAM based.

2.5 Image Processing Algorithms

Many types of image processing algorithms can be embedded within the camera,
since the video processing modules are completely in-system programmable. As an
example, a static grey scale thresholding algorithm is presented below.

2.5.1 Static Grey Scale Thresholding

In static thresholding, an upper and a lower bound are established around what is
considered a normal value. Image data that falls within the boundary window is
considered normal non- interesting data. Image data that falls either above or
below the boundary window is considered data of interest. Considering we are
dealing with an 8-bit digital camera, the normal, upper and lower boundary values
are seen to be digital numbers (DN) on a scale of 0 to 255 "Gray scale". Imagine
that a product is being inspected for defects and the grey scale level of non-
defective product is 85 DN, and the upper and lower boundary values have been
set to +/- 15 DN. All image data that fell within the bounds of 70 DN to 100 DN
would be considered non-interesting and would not be transmitted out of the
camera. Image data that fell below 70 DN and above 100 DN would be considered
interesting and would

be transmitted out of the camera, Substantial data reduction is achieved since only
some of the data will fall outside of the established boundaries. It is important to
note that all of the 'data of interest' is transmitted out of the camera and thus data
reduction is achieved where all of the grey scale information is preserved. This
type of algorithm is illustrated by the image shown in figure 2.7.

Page no.14
Fig 2.7. An Example of Static Thresholding

For later display and analysis each pixel must be given an address such that an
image can be reconstructed by the frame grabber since an algorithm of this type
produces noncontinuous data. The static thresholding algorithm requires three
parameters as follows an upper bound, a lower bound, and a centre value. The
determination of the centre value is essential to this type of algorithm, and the
acceptable band between the upper and lower bound. The static thresholding
algorithm is expressed as follows:

IF (PIXEL GRAY IS> (CENTER + UPPER)) OR (PIXEL GRAY IS (CENTER-

LOWER))
THEN
TRANSMIT PIXEL ELSE
IGNORE PIXEL

2.5.2 Embedded Image Processing Algorithms

The algorithms are embedded in hardware with a PLD/microcontroller

combination and operate at a 20MHz data rate per channel. The effective
processing rate is 40MHz because each image processing PLD can process two
channels of image data. With dedicated DSP controllers such data processing rates
could be difficult to achieve. Microcontroller also can directly control the
algorithm without host computer intervention, since it has access to the image data.

Page no.15
2.6 Smart Camera System

A vision system for web inspection is presented below where a maximum of

twenty 2048 pixel high sensitivity line scan smart cameras are networked together
to a single host computer and frame grabber. A block diagram of the system is
shown in figure 2.8

Page no.16
Fig 2.8. System Block Diagram

The system shown consists of up to twenty 2048 pixel high sensitivity line scan
smart cameras housed within a camera enclosure mounted above the web.
Transmissive illumination is provided since illumination source is mounted
beneath the web. Routed through two cabinets are the data. control, and power
lines to/from the cameras. The system makes use of fibre optic technology for
transmission of data and control signals thus allowing the inspector station to be
located remotely at a distance of up to 100m.

Page no.17
3.Embedded Systems

Today embedded technology produces Intelligent,

Multitasking, Compact electronic products with networking
Facilities

Page no.18
3.1 Introduction to embedded systems:

An embedded system is a special-purpose computer system designed to perform

one or a few dedicated functions, often with real-time computing constraints. It is
usually embedded as part of a complete device including hardware and mechanical
parts. In contrast, a general-purpose computer, such as a personal computer, can do
many different tasks depending on programming. Embedded systems control many
of the common devices in use today.

Since the embedded system is dedicated to specific tasks, design engineers can
optimize it.reducing the size and cost of the product, or increasing the reliability
and performance. Some embedded systems are mass-produced, benefiting from
economics of scale.

Physically, embedded systems range from portable devices such as digital watches
and MP4 players, to large stationary installations like traffic lights, factory
controllers, or the systems controlling nuclear power plants. Complexity varies
from low, with a single microcontroller chip, to very high with multiple units,
peripherals and networks mounted inside a large chassis or enclosure.

In general, "embedded system" is not an exactly defined term, as many systems

have some element of programmability. For example, Handheld computers share
some elements with embedded systems such as the operating systems and
microprocessors which power them-but are not truly embedded systems, because
they allow different applications to be loaded and peripherals to be connected.

Embedded computing systems

Computing systems embedded within electronic devices

Hard to define. Nearly any computing system other than a desktop computer
Billions of units produced yearly, versus millions of desktop units
Perhaps 50 per household and per automobile.

Page no.19
Fig 3.1 An Embedded System

3.2 Examples of embedded systems

Embedded systems span all aspects of modern life and there are many examples of
their use.

Telecommunications systems employ numerous embedded systems from telephone

switches for the network to mobile phones at the end-user, Computer networking
uses dedicated routers and network bridges to route data.

Consumer electronics include personal digital assistants (PDAs), mp3 players,

mobile phones, videogame consoles, digital cameras, DVD players, GPS receivers,
and printers. Many household appliances, such as microwave ovens, washing
machines and dishwashers, are including embedded systems to provide flexibility,
efficiency and features. Advanced HVAC systems use networked thermostats to
more accurately and efficiently control temperature that can change by time of day
and season Home automation uses wired- and wireless-networking that can be used
to control lights, climate, security, audio/visual, surveillance, etc., all of which use
embedded devices for sensing and controlling.

Page no.20
Transportation systems from flight to automobiles increasingly use embedded
systems. New airplanes contain advanced avionics such as inertial guidance
systems and GPS receivers that also have considerable safety requirements.
Various electric motors brushless DC motors induction motors and DC motors are
using electric/electronic motor controllers. Automobiles. electric vehicles, and
hybrid vehicles are increasingly using embedded systems to maximize efficiency
and reduce pollution. Other automotive safety systems such as anti-lock braking
system (ABS), Electronic Stability Control (ESC/ESP), traction control (TCS) and
automatic four-wheel drive.

Medical equipment is continuing to advance with more embedded systems for vital
signs monitoring, electronic stethoscopes for amplifying sounds, and various
medical imaging (PET. SPECT. CT. MRI) for non-invasive internal inspections.

In addition to commonly described embedded systems based on small computers,

a new class of miniature wireless devices called motes are quickly gaining
popularity as the field of wireless sensor networking rises. Wireless sensor
networking, WSN, makes use of miniaturization made possible by advanced IC
design to couple full wireless subsystems to sophisticated sensor,enabling people
and companies to measure a myriad of things in the physical world and act on
this information through IT monitoring and control systems. These motes are
completely self contained, and will typically run off a battery source for many
years before the batteries need to be changed or charged. An Embedded System is
a combination of Hardware and Software thatmay have some mechanical
components to perform specific tasks.

Fig 32 PC Engines ALIX.IC Mini-ITX embedded board with an x86 AMD Geade
LX 800 together with Compact Flash, miniPCI and PCI slots, 22-pin IDE interface,
audio, USB and 256MB RAM

Page no.21
3.3 Characteristics of embedded systems:

1. Device programmability or manageable-

The functioning of a specifically designed hardware part on a smart device can

easily be changed by simply changing the software associated with it.

2. Multitasking-

An embedded system employed in a modern refrigerator performs door sense and

temperature sense at the same times, which are two functions at the same time.

3. Real time response-

It is the ability of an embedded system to respond to ambient conditions suddenly.

That is, a smart TV adjusts picture quality suddenly in response to sudden
environmental brightness variations.

4. Embedded systems are designed to do some specific task, rather than be a

general- purpose computer for multiple tasks. Some also have real-time
performance constraints that must be met, for reasons such as safety and usability;
others may have low or no performance requirements, allowing the system
hardware to be simplified to reduce costs.

5. Embedded systems are not always standalone devices. Many embedded systems
consist of small, computerized parts within a larger device that serves a more
general purpose. For example, the Gibson Robot Guitar features an embedded
system for tuning the strings, but the overall purpose of the Robot Guitar is, of
course, to play music.[2] Similarly, an embedded system in an automobile provides
a specific function as a subsystem of the car itself.

6. The program instructions written for embedded systems are referred to as

firmware, and are stored in read-only memory or Flash memory chips. They run
with limited computer hardware resources: little memory, small or non-existent
keyboard and/or screen

Page no.22
4.Smart Camaras As Embedded Systems
A video camera coupled to a computer vision system is
a tiny package of embedded systems.
&
Co working of hardware and software engineers is the basis of Embedded
Application Development.

Page no.23
4.1 Overview of Smart Cameras in embedded systems

The system also provides input and output to the plant control system. Data from
the cameras is acquired by the frame grabber, assembled into images, and then
transferred to the host computer real time display on the defect monitor, and stored
to a database residing on the file server via an Ethernet connection. Subsequent
analysis of the data is performed at the analysis workstation with analysis software
that allows extraction of data from the database for creation of reports. All system
and analysis software is multithreaded and provides real time data access and
display. Via a modem connection the system is also operable remotely. To ensure
smooth and constant illumination of the web the system software also controls the
illumination source with a fuzzy logic control scheme.

Recent technological advances are enabling a new generation of smart cameras that
represent a quantum leap in sophistication. While today's digital cameras capture
images, smart cameras capture high-level descriptions of the scene and analyze
what they see. These devices could support a wide variety of applications including
human and animal detection, surveillance, motion analysis, and facial
identification

Because they push the design space in so many dimensions, smart cameras are a
leading edge application for embedded system research.

4.2 Detection and Recognition Algorithms

Although there are many approaches to real-time video analysis, we chose to focus
initially on human gesture recognition identifying whether a subject is walking,
standing, waving his arms, and so on. Because much work remains to be done on
this problem, we sought to design an embedded system that can incorporate future
algorithms as well as use those we created exclusively for this application
Our algorithms use both low-level and high-level processing. The low-level
component identifies
different body parts and categorizes their movement in simple terms. The high-
level component, which is application-dependent, uses this information to
recognize each body part's action and the person's overall activity based on
scenario parameters. Human detection and activity/gesture recognition algorithm
has two major parts: Low-level processing (blue blocks in Figure 4.1) and high-
level processing (green blocks in Figure 4.1),
Page no.24
A) Low-level processing

The system captures images from the video input, which can be either
uncompressed or compressed (MPEG and motion JPEG), and applies four different
algorithms to detect and identify human body parts.

Region extraction:

The first algorithm transforms the pixels of an image like that shown in Figure 4.2
a, into an M¥ N bitmap and eliminates the background. It then detects the body
part's skin area using a YUV color model with chrominance values down sampled

Page no.25
Figure 4.2 a

Next, as Figure 4.2 b illustrates, the algorithm hierarchically segments the frame
into skin tone and non-skin-tone regions by extracting foreground regions adjacent
to detected skin areas and combining these segments in a meaningful way.

Figure 4.2 b

Page no.26
Contour following:
The next step in the process, shown in Figure 4,2 c, involves linking the separate
groups of pixels into contours that geometrically define the regions. This algorithm
uses a 33 filter to follow the edge of the component in any of eight different
directions.

Figure 4.2 c

Ellipse fitting:

To correct for deformations in image processing caused by clothing, objects in the

frame, or some body parts blocking others, an algorithm fits ellipses to the pixel
regions as Figure 4.2d shows to provide simplified part attributes. The algorithm
uses these parametric surface approximations to compute geometric descriptors for
segments such as area, compactness (circularity), weak perspective invariants, and
spatial relationships.

Page no.27
Graph matching:

Each extracted region modeled with ellipses corresponds to a node in a graphical

representation of the human body. A piecewise quadratic Bayesian classifier uses
the ellipses parameters to compute feature vectors consisting of binary and unary
attributes. It then matches these attributes to feature vectors of body parts or
meaningful combinations of parts that are computed offline. To expedite the
branching process, the algorithm begins with the face, which is generally easiest to
detect.

B) High-level processing

The high-level processing component, which can be adapted to different

applications, compares the motion pattern of each body part-described as a
spatiotemporal sequence of feature vectors -in a set of frames to the patterns of
known postures and gestures and then uses several hidden Markov models in
parallel to evaluate the body's overall activity. We use discrete HMMs that can
generate eight directional code words that check the up, down, left, right, and
circular movement of each body part.
Human actions often involve a complex series of movements. We therefore
combine each body part's motion pattern with the one immediately following it to
generate a new pattern. Using dynamic programming, we calculate the
probabilities for the original and combined patterns to identify what the person is
doing. Gaps between gestures help indicate the beginning and end of discrete
actions.
A quadratic Mahalanobis distance classifier combines HMM output with different
weights to generate reference models for various gestures. For example, a pointing
gesture could be recognized as a command to "go to the next slide" in a smart
meeting room or "open the window in a smart car, whereas a smart security camera
might interpret the gesture as suspicious or threatening.
To help compensate for occlusion and other image-processing problems, we use
two cameras set at a 90-degree angle to each other to capture the best view of the
face and other key body parts. We can use high-level information acquired through
one view to switch cameras to activate the recognition algorithms using the second
camera. Certain actions, such as turning to face another direction or executing a
predefined gesture, can also trigger the system to change views Soft-tissue
reconstruction.
We can use MatLab to develop our algorithms. This technical computation and
visualization programming environment runs orders of magnitude more slowly
than embedded platform implementations, a speed difference that becomes critical
when processing video in real time. We can therefore port our MatLab
implementation to C code running on a very long instruction word (VLIW) video
processor, which lets us make many architectural measurements on the application
and make the necessary optimizations to architect a custom VLSI smart camera.
Page no.28
4.3 Requirements

At the development stage, we can evaluate the algorithms according to accuracy

and other familiar standards. However, an embedded system has additional real-
time requirements:

Frame rate:

The system must process a certain amount of frames per second to properly
analyze motion and provide useful results. The algorithms we use as well as the
platform's computational power determine the achievable frame rate, which can be
extremely high in some systems.

Latency:

The amount of time it takes to produce a result for a frame is also important
because smart cameras will likely be used in closed-loop control systems, where
high latency makes it difficult to initiate events in a timely fashion based on action
in the video field.

Moving to an embedded platform also meant that we have to conserve memory.

Looking ahead to highly integrated smart cameras, we want to incorporate as little
memory in the system as possible to save on both chip area and power
consumption. Gratuitous use of memory also often points to inefficient
implementation

4.4 Components

Our development strategy calls for leveraging off-the-shelf components to process

video from a standard source in real time, debug algorithms and programs, and
connecting multiple smart cameras in a networked system. We use the 100-MHz
Philips TriMedia TM-1300 as our video processor. This 32-bit fixed- and floating-
point processor features a dedicated image coprocessor, a variable length decoder,
an optimizing C/C++ compiler, integrated peripherals for VLIW concurrent real-
time input/output, and a rich set of application library functions including MPEG,
motion JPEG, and 2D text and graphics

Page no.29
4.5 Testbed Architecture

Our testbed architecture, shown in Figure 4.3, uses two TriMedia boards attached
to a host PC for programming support. Each PCI bus board is connected to a His
camera that provides NTSC composite video. Several boards can be plugged into a
single computer for simultaneous video operations. The shared memory interface
offers higher performance than the networks likely to be used in VLSI cameras,
but they let us functionally implement and debug multiple-camera systems with
real video data.

Fig 4.3 Testbed architecture

4.6 Experiments and Optimizations

As data representation becomes more abstract, input/output data volume decreases.

The change in required memory size, however, is less predictable given the
complex relationships that can form between abstract data. For example, using six
single precision, floating-point parameters to

describe 100 ellipses requires only 2.4 Kbytes of memory, but it takes 10 Kbytes to
store information about two adjoining ellipses. Based on these early experiments,
we optimize our smart camera implementation by applying techniques to speed up
video operations such as substituting new algorithms better suited to real time
processing and using TriMedia library routines to replace Clevel code.

Page no.30
4.7 Algorithmic changes

We originally fit super ellipses (generalized ellipses) to contour points, and this
was the most time-consuming step. Rather than trying to optimize the code, we
decided to use a different algorithm. By replacing the original method developed
from principal component analysis with moment-based initialization, we reduced
the Levenberg- Marquardt fitting procedure, thus decreasing the execution time.
After converting the original Matlab implementation into C, we performed some
experiments to gauge the smart camera system's effectiveness and evaluate
bottlenecks. The unoptimized code took, on average, 20.4 million cycles to process
one input frame, equal to a rate of 5 frames per second.
We first measure the CPU times of each low-level processing step to determine
where the cycles were being spent. Microsoft Visual C++ is more suitable for this
purpose than the TriMedia compiler because it can collect the running time of each
function as well as its sub functions'
times. Figure 4.4a shows the processing-time distribution of the four body-part-
detection algorithms
Figure 4.4b shows the memory characteristics of each low-level processing stage.

Figure 4.4 a, 4.4b

Page no.31
4.8 Control-to-data transformation

Increasing the processor's issue width can exploit the high degree of parallelism
that region extraction offers. Using a processor with more functional units could
thus reduce processing time during this stage. However, contour following, which
converts pixels to abstract forms such as lines and ellipses, consumes even more
time. The algorithm also operates serially: It finds a region's boundary by looking
at a small window of pixels and sequentially moving around the contour; at each
clockwise step it must evaluate where to locate the contour's next pixel. While this
approach is correct and intuitive, it provides limited ILP.

We evaluate all possible directions in parallel and combined the true false results
into a byte.

Page no.32
5.Applications Of Smart Cameras

Application of smart cameras takes place where volume, pricing or reliability

constraints forbid use of bulkier devices and PC's.

Page no.33
5.1 Fields of application:

Smart cameras can in general be used for the same kind of applications where
more complex vision systems are used, and can additionally be applied in some
applications where volume, pricing or reliability constraints forbid use of bulkier
devices and PC's

Typical fields of application are:

• Vision-based smart environments.

• Surveillance and tracking applications. • Applications based on fusion of
visual and other sensory data.
• Multi-view vision for Human-Computer Interfaces (HCI).
• 3D scene analysis with distributed sensors. • Distributed multimedia and
gaming applications.
• Automated inspection for quality assurance (detection of defects, flaws,
missing parts...).
• Non contact measurements.
• Part sorting and identification.
• Code reading and verification (barcode. Data Matrix, alphanumeric etc.)
• web inspection (inspection of continuously flowing materials such as coils,
tubes, wires, extruded plastic) for defect detection and dimensional gauging.
• detection of position and rotation of parts for robot guidance and automated
picking
• unattended surveillance (detection of intruders, fire or smoke detection)
biometric recognition and access control (face, fingerprint, iris recognition)
• visual sensor networks

Developers can purchase smart cameras and develop their own programs for
special, custom made applications, or they can purchase ready made application
software from the camera manufacturer or from third party sources. Custom
programs can be developed by programming in various languages (typically C or
C++) or by using more intuitive, albeit somewhat less flexible, development tools
where existing functionalities (often called tool or blocks) can be connected in a
list (a sequence or a bidimensional flowchart) that describes the desired flow of
operations without any need to write program code. The main advantage of the
visual approach Vs. programming is in a much shorter and somewhat easier
development process, available also to non-programmers.
Page no.34
Smart cameras running software tailored for a single specific application are often
called "vision sensors,"

The term "smart camera" for an image capture system with embedded computing
was used by Reticon for one of their products, by 1977.

5.2 Applications of Smart Camera Networks

Self localizing smart camera networks can serve as an enabling technology for a
wide range of higher level applications. Here we focus on two applications where
the images from the camera systems are used to derive information about the
geometric structure of the environment.

5.2.1 Visual Hull Reconstruction

Multi camera systems are commonly used to derive information about the three
dimensional structure of a scene. One approach to the reconstruction problem
which is particu larly well suited to the proposed self localizing smart camera
network is the method of volume intersection which has been employed in various
forms by a number of researchers. This method can be used to detect and localize
dynamic objects moving through the field of view of the smart camera network.
Here a set of stationary cameras are used to observe one or more objects moving
through the scene. Simple background subtraction is employed to delineate the
portions of the images that correspond to the transient objects. Once this has been
accomplished one can interrogate the occupancy of any point in the scene, P, by
projecting it into each of the images in turn and determining whether or not it lies
within the intersection of the swept regions. This process can be used to produce an
approximation for the 3D structure of the transient objects by sampling points in
the volume. The results of such an analysis are shown in Figure5.1.

Figure 5.1(a) Background image of a scene 5.1(b) Image with object inserted
Page no.35
5.1c) Results of the background subtraction operation 5.1(d) Results of applying
the volumetric
reconstruction procedure to the difference images derived from the three smart
camera nodes.

In this application the ability to rapidly localize a set of widely separated cameras
is a distinct advantage. Other implementations of this reconstruction scheme
involve complex, time consuming calibration operations. This implementation, in
contrast, could be be quickly deployed in an ad-hoc manner and would allow a user
to localize and track moving objects such as people, cars or animals as they move
through the scene.

Page no.36
6.Conclusion And Some Future Trends

A Musial fountain dances according to tunes of a music-this is the smart

world created by embedded technology.
In future, Smart Cameras will be able analyze and understand scenes they
observe in a cooperative manner.

Page no.37
6.1 Optimization results and Conclusion

The combination of these methods radically improves CPU performance for the
application. Optimization boosts the program's frame rate from 5 to 31 frames per
second. In addition, latency decreases from about 340 to 40-60 milliseconds per
frame. We can add HMMs and other high-level processing parts, and that makes
the program now runs at about 25 frames per second. Our board-level system is a
critical first step in the design of a highly integrated smart camera. Although the
current system is directly useful for some applications, including security and
medicine, a VLSI system will enable the development of high-volume, embedded
computing products.

Because the digital processors and memory use advanced small-feature fabrication
and the sensor requires relatively large pixels to efficiently collect light, it makes
sense to design the system as two chips and house them in a multichip module.
Separating the sensor and the processor also makes sense at the architectural level
given the well understood and simple interface between the sensor and the
computation engine.

The advantages of leveraging existing sensor technology far outweigh any benefits
of using pixel-plane processors until they become more plentiful. However,
attaching special-purpose SIMD processors to the multiprocessor can be useful for
boundary analysis and other operations Such accelerators can also save power,
which is important given the cost and effort required to deploy multiple cameras,
especially in an outdoor setting. High-frame-rate cameras, which are useful for
applications ranging from vibration analysis to machinery design, will likely
require many specialized processing elements that are fast as well as area efficient.

In contrast to common digital cameras, as known for example for taking holiday
pictures. Smart Cameras are equipped with a computing unit that is used for the
analysis of captured images. Recent advances in the research field of Computer
Vision allow for the detection of human faces and gestures and for the
measurement of distances between objects. In future, Smart Cameras will be able
analyse and understand scenes they observe in a cooperative manner. Distributed
Smart Cameras can be connected to each other by ad-hoc or infrastructure
networks. This Distributed System does not rely on a central control console and
has therefore no single point of failure - an important aspect when safety and
robustness are concerned.

Page no.38
This paper describes a scheme for determining the relative location and orientation
of a set of smart camera nodes and sensor modules. The scheme is well suited for
implementation on wireless sensor networks since the communication and
computational requirements are quite minimal. Self localization is a basic
capability on which higher level applications can be built. For example, the scheme
could be used to survey the location of other sensor motes enabling a range of
location based sensor analyses such as sniper detection, chemical plume detection
and target tracking.

Further, the ability to automatically localize a set of smart cameras deployed in an

ad-hoc manner allows us to apply a number of multi-camera 3D analysis
techniques to recover aspects of the 3D geometry of a scene from the available
imagery. Ultimately we envision being able to construct accurate 3D models of
extended environments based on images acquired by a network of inexpensive
smart camera systems.

6.2 Looking Ahead

The vision industry is rapidly moving away from the video camera/frame grabber
systems of the twentieth century to a new generation of smart-camera-based
systems for the 21" century. These 21st century smart-camera systems will perform
real-time, pixel-data extraction and processing operations within the camera at
extremely high speeds and at a cost, which is considerably less than required today
for comparable capabilities. Eventually, complete vision-processing-systems- on-a-
sensor-chip will be available.

Components in smart cameras will undoubtedly change due to the push from
semiconductors and new microprocessors coming onto the market. The trends for
the next year will be towards megapixel sensors, higher resolution, faster
processing power, and colour. A typical standard CCD based camera has a matrix
of 480,000 pixels, however the new megapixel cameras offer at least 1.000x1,000
pixels or million pixels. Some manufacturers already offer cameras of 2 million
pixels.

6.2.1 Future Applications

To date, exploitation of smart camera technology has been mainly for industrial
vision systems,
Page no.39
but a crossover is just starting to take place. Smart camera technology will begin to
enter new applications, for example, in the security and access control markets, in
the automotive industry. for collision avoidance, and even one day for the toy
industry. Even our automobiles may soon be outfitted with miniature eyes. Built
into a cruise control system, for instance, such a camera would suddenly alert the
driver if it noted a rapidly decelerating vehicle. The cameras could also take the
place of the rear view and side-view mirrors, thereby eliminating dangerous blind
spots and in the event of an accident-recording the seconds prior to a collision
Another example would be with intelligent lifts. An office block, with many lifts
and floors, may see a lot of people travelling up and down between floors,
particularly at high traffic times such as early morning or end of the working day.
At the moment, lifts are called by somebody pressing a button and putting in a
request for the lift to stop at a particular floor. Connected with smart camera
technology, lifts could be routed on demand, working intelligently, stopping only
when there was a pre-set number of passengers waiting at a floor-and missing out a
floor if too many people were waiting to meet the maximum capacity of the lift.
Looking into the future, we can foresee an infinite number of applications for the
smart camera; in fact, as many as there are potential image processing uses.
Distributed Smart Cameras can be used for a wide range of applications. To name
a few, the following:

1.Assisted Living, Health-Care:

Our society is aging constantly. This leads to a large number of elderly people that need to be
cared for. Many elderly move to residential homes because they are frightened of accidents that
might occur at home and leave them helpless without the possibility to call for help. Smart
Cameras can detect accidents on their own and inform either relatives or a mobile nursing
service.

2.Surveillance of large, safety-critical areas (airports, train stations):

Common surveillance systems are prone to errors since human staff in central control rooms can
usually not analyse all video data captured by the surveillance cameras. Critical events may pass
unnoticed and cause great harm that could have been avoided by the use of Smart Cameras.
Smart Cameras are able to detect critical events (e.g. persons leaving suspicious luggage or
entering forbidden areas) on their own and raise an alarm. Notifications about suspicious
incidents can be sent directly from a Smart Camera to a PDA carried by security staff nearby.

3.Industry:
Common sensors used for automation often rely on wired, electromechanical devices. The use of
wireless, contactless vision-based devices can help to reduce costs, since maintenance costs of
common sensors are high.
Page no.40
REFERENCE

[1]. Wayne Wolf, Burak Ozer, Tiehan Lv, "Smart Cameras as High-Performance
Embedded Systems".

[2]. Burak Ozer, Wayne Wolf, "A Hierarchical Human Detection System in
Compressed and Uncompressed Domains".

[3]. Tiehan Lv. Burak Ozer. Wayne Wolf, "Smart Camera System Design." Invited
Paper. International Packet Video Workshop. Pittsburgh, April 2002.

[4] Advanced Imaging Europe Magazine: The intelligent camera, October (2002)
12-16

[5] Wintriss Engineering Corporation: Year 2002 Smart Cameras

[6] Smart Cameras vs. PC-based Machine Vision Systems,

http://www.coreco.com/, (2002)

[7] Longbottom, D.: Latest Developments in Sensor and Camera Technology.

Alrad Instruments

Ltd., White paper, http://www.ukiva.org/IPOTMV02Latest.pdf. (2002) [8] Smart

Cameras - A complete vision system in a camera body. Vision Components,

http://www.is.irl.cri.nz/products/smartcam.html

[9] Lehotsky, D. A.: Intelligent High Sensitivity CCD Line Scan Camera with
embedded Image Processing algorithms, DALSA INC, (2002)

[10] Siemens R&1: The Camera that Grew a

Brain,http://w4.siemens.de/Ful/en/archiv/zeitschrift/

heft1_99/artikel02/

[11] http://en.wikipedia.org/wiki/Smart camera

[12] http://www.icdsc.org/
Page no.41