Color Detection Using FPGA Based Smart Camera
Color Detection Using FPGA Based Smart Camera
Table of Contents
1 2 3 Abstract Introduction Project Description 3.1 CMOS Camera D5M 3.2 FPGA Cyclone II 3.3 LTM 3.4 Clock Issues 3.5 Verilog Implementation Project Functionality 4.1 CMOS Pixel Capture 4.2 RAW Data to RGB 4.3 SDRAM Controller 4.4 RGB Video 4.5 Grayscale Video 4.6 Sobel Filter 4.7 Color detector in Sobel edge detection technique 4.8 Orange Color Detector 4.9 Dilation 4.10 Display Controller Platform 5.1 Hardware 5.2 Software Test Results Applications Conclusion 3 4 5 5 6 6 6 6 8 9 9 11 11 11 12 13 15 16 16 16 16 16 17 18 18
6 7 8
1. Abstract:
Object recognition and tracking with real-time smart camera plays an important role in many modern vision applications such as work force tracking, intelligence surveillance, fire detection and many more. The main motivation of this project is to extract the color object and recognition using the Sobel edge detection and pass the detected signal to wireless channel using Zigbee (802.15.4). Experimental results demonstrated that object detection is very important for selected color tracking in the industrial automation and work places. Computer based Software programming such as java, C++ and C# cannot satisfy the speed in terms of real-time parallel processing of vast amount of video data. Therefore in order to improve the performance of real time video processing, FPGA is an effective device to rely on. This device is capable of real-time parallel processing of vast amounts of video data. The presented results from given design can be used as color and edge detection on a streaming video which is the basic building block necessary for further object detection. For detection of edge, various detection operators are used. Most of these operators are applied with convolution masks and are based on differential operations.
2. Introduction:
The idea of this project is to design and develop a stand-alone, FPGA-based smart camera that can perform the processing tasks such as Sobel, Prewitt, dilation edge detection, color object detection, emboss effects and output the alarm signal in the case of detection to short-range wireless systems using the zigbee. With the rapid development of remote sensing camera has become a popular sensor device for monitoring and surveillance systems. It is the first step of video analysis and understanding with remote sensing video especially the detection of object which is necessary step to extract the information from stream of binary raw data from the sensors. The purpose of the detection is to discover the information about the identified object in the presence of the surrounding objects. The sensor device used is a THDB-D5M, which is five mega pixel digital CMOS color image sensor and it contains a power voltage circuit, a low noise amplifier and clock driver circuit. Its flexibility and ease of use makes it a good choice for the system. The raw data of color video stream and edge detection is very large, so speed of video processing is a difficult problem. Therefore in order to improve the performance of real time video processing FPGA is an effective device to realize real-time parallel processing of vast amounts of video data. Moreover, adopting FPGA as a development platform than PC and other IC is to perform data intensive video processing tasks with real requirements. Therefore, architecture of detection is designed with a FPGA chip called Cyclone II 2C70, and it can process 800 x 480 x 10 RGB color scale video successfully. This device can locate the detected color object and the edge of the gray frame quickly and efficiently.
3. Project description:
The basic principle of this project is shown in the fig 4.The system architecture is divided into main three parts: an image capture unit (D5M camera), an FPGA-based signal processing unit (Cyclone II), and a display unit (LTM).
Light Sensor
Fig 3: Pixel array description
Red Channel Green Channel Blue Channel Vertical Sync Horizontal Sync Data Enable
Display Controller
Signal Processing
Display
A typical camera system is shown in figure 1. At the heart of every digital camera is an image sensor either CCD image senor or a CMOS sensor. Both types of sensor capture the light through the lens and convert into the electrical signal which is further processed by ADC. Nowadays, majority of sensors consist of high performance ADC converters which are employed to produce the digital output. Most CMOS image sensors have ADC resolutions of 10 to 12 bits. In this project, D5M camera has 12 bits ADC resolution. The D5M pixel array consists a 2752-coloumn by 2004-row matrix. The pixel array is addressed by column and row. The array consists of a 2592-column by 1944-row active region and a border region as shown below.
The output images are divided into frames, which are further divided into lines. The output signals frame valid and lines valid are used to indicate the boundaries between the lines and frames [3]. Pixel clock is used as a clock to latch the data. For each pixel clock cycle, one 12-bit pixel data outputs on the data out pins [3]. When both frame valid and line valid are asserted, the pixel is valid. Pixel clock cycles that occur when frame valid is negated are called vertical blanking. Pixel clock cycles that occur when only line valid is negated are called horizontal blanking. The camera has an array of 255 register, a lot of them are
6 configurable, and it is possible to set up the operation of the camera by writing to these registers. This communication is performed using a generic two wire serial interface commonly referred to as I2C[3]. There are two different communication modes, control / configuration mode which included read/write values to the register in the D5M card, and pixel data read out which consists of reading the pixel data from the camera card [3].
Redesign/Re-spec
Wont Synthesis
Pixel Capture
Once Orange color is detected, and then output the alarm signal to wireless channel using USB port.
Detected
Not Detected
External SDRAM
32 bit Bus
The user has the option to change between the edges detections to color and gray video. Also the user can change the edge detection threshold.
Orange Color Detection with Specified Threshold of RGB ((R>230 & R<255), (G>30 & G<150), (B>30 & B<150))
Edge Detection
Color Detection
800
Fig 5: Video Color Object and Edge Detection Top-level Hardware Structure Design of the proposed system
Input clock 96 MHz (XCLKIN) Input frame valid and line valid Latched data at 50MHz (PIXCLK) 12 Bits DATA/Pixel at the Max data of 96Mp/s
always@(posedge clockat50Mhz or negedge Reset) begin Red<=Data1; Green<=Data0 +Data1; Blue<=Data0; end
RAW TO RGB
12 Bits Serial Data
Raw image from camera Colors are stripped out to Bayer RGB
Bayer Pattern
+
Fig 7: I/O Timing Characteristics
11
12
st
Kvy1 Kvy3
Fig 9: 3-Line buffer connecting 3x3 Filter for vertical Sobel Edge Detection and output from the Mac
Three lines video buffer have used for a filter with 3x3 kernel for both vertical and horizontal edge detection .Then the video buffer enters the filter in a parallel form. Each 3 t-taps module takes the data taps from the line buffer with a rate faster than system clock rate by 3 times and multiply them by the 3 filter coefficients. The output from each multiplications process is accumulated in the accumulator. Then finally sum all three 3-Tap filter from the accumulator fed into parallel adder which form one processed pixel. Once the Rxy is calculated the value is compared with threshold which is the user input from the switches (12 to 1).If the pixel output from Rxy is greater than the threshold, the pixel is detected as the edge otherwise no output to the RGB channel. The output from the sobel channel is fed into the red, green and blue channel which is further processed for detection in the detection module. Sobel edge detection
13 module interface with the Display controller and the color detector module through the mux of switches when [SW#13 Low, SW#14Low, SW#15 High] are selected.
14 Normal RGB stream is displayed on the screen P(RGB in) =The Whole Frame Pixels within the Region (Color Detected): P ((RGB)in (RGB)window) = True (overlap (RGBout=RGBin)) Pixels not within the Region (Sobel Detection): P ((RGB)in (RGB)window) = False (not overlap(RGBout=RGBin))
Predicate P defines the region which involves RGB pixel values or area feature.
15
16
4.9 Dilation:
Dithering is used to transform a grayscale video into binary video, while preserving some grayscale information. Each grayscale value of the original video is translated into a pattern of binary values in the output image.
Before RGB data enters this module, the user has the option to select the desired detection from the parallel channels of RGB which is to be displayed on the screen. More functionally of the threshold is enabled through the 12 switches [12:1].
Detection:
Gray Video with color Object Detection Prewitt Edge Detection Cartoon Effect Dilation Edge Detection Sobel Edge Detection Sobel with color Object Detection RGB Video with Sobel Edge Detection Emboss Effect
Switches
SW15=Low , SW14=Low, SW13=Low SW15=Low , SW14=Low, SW13=High SW15=Low , SW14= High, SW13=Low SW15=Low , SW14= High, SW13= High SW15= High, SW14=Low, SW13=Low SW15= High, SW14=Low, SW13= High SW15= High, SW14= High, SW13=Low SW15= High, SW14= High, SW13= High
17
5. Platform:
5.1 Hardware:
Hardware used for this projects are DE2-70 board, LTM display, D5M camera and two zigbee.
5.2 Software:
A wide varity of software are used for this project, such as Quartus [4], Synplify [5], ModelSim [6] and XCTU [7]. Quartus is programmable logic device design software from Altera. Its features include: An implementation of VHDL and Verilog for hardware description. Visual edition of logic circuits. Vector waveform simulation. ModelSim is used to simulate and verify the functionality of verilog code. Synplify is the primary tool used to create the hardware design which gives the RTL view to visualize the design and to find the errors in advance. The ModelSim computer software is a great tool for debugging the code. To enable the direct XBee communication, Digi's X-CTU application is used to read and write the settings of XBee module.
18
6. Test Results:
Filter Results:
19
7. Applications:
Defective items are identified and prevented from reaching to the customer. Following are the applications of using this device in the industry. Smart camera can used to provide better stability in measurement. The camera performs RGB processing of marks, such as print on cardboard, half-filled drink bottle to reliably detect subtle differences.
GO
NOGO
Fig 13: The color of print on cardboard is easily registered and identified as RGB information.
GO
NOGO
Fig 15: The color of bottle is easily registered and identified as RGB information.
8. Conclusion:
Overall the results from the project were satisfied. In this paper, we have shown how to create a pixel which is based on color window detector. We elaborated the concept of edge detection using FPGA device. Improvement can be made for the future work with more in depth parallel and pipeline structure using low level programming techniques. Finally, the least the color detection of any range in FPGA chip also inspires more intelligent control of the system in the future. This stand-alone, FPGA-based smart camera is useful in the tracking of a moving object in the factories, workers vest, operator position and missing labels on the object.
References:
[1] http://nptel.iitm.ac.in/courses/Webcoursecontents/IIT%20Kharagpur/Embedded%20systems/Pdf/Lesson-20.pdf [2] http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=5716147 [3] http://www.secs.oakland.edu/~ganesan/ece576f10project/index_files/Page361.htm [4] Quartus FPGA design software, from http://www.altera.com/products/software/quartus-ii/web-edition/qts-we-index.html [5] The Synplify solution Synplicity, from http://www.synplicity.com/products/synplifypro/ [6] Mentor Graphics Modelsim, from http://www.mentor.com/products/fpga/simulation/modelsim [7] http://www.digi.com/support/kbase/kbaseresultdetl.jsp?kb=125