Super-Resolution Based on Open Source Computer Vision Library EmguCV

Lee Jee Shen

Faculty of Engineering and Technology Multimedia University Malacca, Malaysia [email protected]
AbstractIn this paper, we present a simple implementation of Super-Resolution technique using EmguCV computer vision library. Instead of static images, we use real-time video inputs from web camera as the input. Each Super-Resolution image result are processed based on multiple frames of the video input. The Super-Resolution technique applied our application is implemented with the following sequence image acquisition, image registration, sub-pixel image shift calculation, image transformation and lastly image interpolation. Index TermsSuper-Resolution, EmguCV, real-time video input image processing

Authors Name/s per 2nd Affiliation (Author)

Dept. name of organization (Line 1 of Affiliation - optional) Name of organization - acronyms acceptable (line 2) City, Country (line 3) [email protected] optional (line 4)

I. INTRODUCTION Super-Resolution is an image processing technique that takes multiple shifted images of a same scene and generate a higher resolution image copy of the scene by considering lost image information that are captured at the other frames of the same scene due to vibration or image shift. The main concept behind Super-Resolution is that CCD camera sensors are finite and discrete while the light signal is continuous. In one image frame taken by CCD camera, there are many lost information of the scene that does not get picked up by the camera sensor grid because some continuous light signal will not hit the CCD sensor grid directly. Figure 1 illustrate the scenario when a full hit of light signal on CCD camera sensor grid while the second image (Fig.2) illustrate the scenario of none full hit (information lost) when the light hit in the might of the sensor grid. EmguCV is an open source computer vision library written on .NET platform. Its a translated copy of OpenCV with some additional feature specifically written for .NET programmer. EmguCV act as a wrapper of OpenCV to access memory level data which was considered as (unsafe) programming in the .NET environment. The library is written using C#.NET programming languages and it can be compiled into Window, Linux, Max OS, Android compatible with the Mono compiler. Mono Mono is a compiler written to compile C# language standard into the target platform. On top of EmguCV, we are using another open source computer vision library AForge specifically to take web camera input.
Fig. 1. CCD sensor are recording the intensity of the light based on the amount of intensity hit the sensor grid.

Fig. 2. Extra information (the tip feature of the object) are better recorded.

II. SUPER-RESOLUTION TECHNIQUE There are many Super-Resolution techniques proposed but generally it involves the following steps image acquisition, image registration, sub-pixel shift estimation, and interpolation. In our application, we started by first acquiring the first five

image frames from web camera and then feed them into the Super-Resolution image processing function that we implemented to get the first Super-Resolution of our video stream. From then onward, we repeatedly update the SuperResolution frame based on the latest video frames acquired from the web camera.

region of an image input for processing. Region of Interested are registered using System.Windows.Rectangle class with the help of MouseDown and MouseUp event.

IV. IMAGE REGISTRATION Image registration is the most important part of SuperResolution because its responsible for the feature detection and also the sub-pixel calculation that determine the quality for the output. To make out Super-Resolution implementation reusable in other application, we create a separate project and wrote our SuperResolutionFilter class. In the class, we defined the following object properties: TABLE I.
Property Name ScaleFactor Fig. 3. Three image video frames (on the left) are taken in this example on the same scene. Based on the feature, sub-pixel shift of the scene are detected on Frame 2 and Frame 3. FramesFusionAccuracy Threshold InputFrames SuperResolutionFrame Roi ShiftResult
CLASS PROPERTY

Description Storing the scaling factor of the target output. Minimum accuracy threshold that the shift result confidence must comply before considering as a valid estimation List of input frames to store the latest frames from web camera. Storing the latest Super-Resolution image result. Storing the Region of Interest of the image. Storing the X and Y shifted pixel in float data type.

Fig. 4. Sub-pixel shift are estimated and corrected before merging into the final result to produce better estimation of the real scene (SuperResolution).

III. VIDEO FRAME ACQUISITION Image acquisition from the web camera are done using the AForge library. VideoCaptureDeviceForm class are used to detect and selected the web camera. After the selection, IVideoSource passed and initialised. The NewFrame event of the IVideoSource are subscribed in the application to grab the latest video frame from web camera and further triggering the Super-Resolution processing. On top of this, we added a feature to define Region of Interest (ROI) to select specific In order to find the sub-pixel of images with reference to the first image, we first find a feature on an image to track. We are using GoodFeaturesToTrack method of EmguCV to detect features on the video inputs. Before the images are pass to GoodFeaturesToTrack it was convert into the right image format required by EmguCV which is Image<Grey, byte>. To get better result, the resulted features detected are further

passed to FindCornerSubPix method of EmguCV to get subpixel accurate corner of the image.

Based on the features found, a sub-pixel shift of important feature are calculated using OpticalFlow.PyrLK function. OpticalFlow.PyrLK calculates the optical flow for a sparse features set (detected in the earlier steps) using iterative LucasKanade method in pyramids. To produce better result, we added some additional code to calculate the confidence level of each calculated shift. This confidence level value will be used later in the merging steps to filtered inaccurate sub-pixel shift estimation.

Fig. 5. The implemented Super-Resolution application using Visual Studio 2013 and EmguCV.

Fig. 6. On the left is the video stream from web camera and on the right is the result stream generated live.

Based on the calculated sub-pixel shift results and confidence level, the images are transforms using WarpPerspective to correct the sub-pixel shift based on the estimation. Lastly, the images list are merge using RunningAvg of EmguCV to produce the Super-Resolution image. The application then repeat the same process describe above on all the incoming video frames except this time using the latest Super-Resolution image calculated in the last iteration as reference to produce better result. V. EXPERIMENTAL RESULTS To test the implemented Super-Resolution technique implemented using EmguCV, we run our application with many test conditions. Below are sample of the results recorded during the test.

Fig. 7. Improvement can be seen on the right-hand generated SuperResolution image as compare to the original image taken from web camera (on the right) when both images are zoom in for inspection.

ACKNOWLEDGMENT The preferred spelling of the word acknowledgment in America is without an e after the g. Avoid the stilted expression, One of us (R. B. G.) thanks . . . Instead, try R. B. G. thanks. Put applicable sponsor acknowledgments here; DO NOT place them on the first page of your paper or as a footnote. REFERENCES List and number all bibliographical references in 9-point Times, single-spaced, at the end of your paper. When referenced in the text, enclose the citation number in square brackets, for example: [1]. Where appropriate, include the name(s) of editors of referenced books. The template will number citations consecutively within brackets [1]. The sentence punctuation follows the bracket [2]. Refer simply to the reference number, as in [3]do not use Ref. [3] or reference [3]. Do not use reference citations as nouns of a sentence (e.g., not: as the writer explains in [1]). Unless there are six authors or more give all authors names and do not use et al.. Papers that have not been published, even if they have been submitted for publication, should be cited as unpublished [4]. Papers that have been accepted for publication should be cited as in press [5]. Capitalize only the first word in a paper title, except for proper nouns and element symbols. For papers published in translation journals, please give the English citation first, followed by the original foreign-language citation [6].
[1] G. Eason, B. Noble, and I. N. Sneddon, On certain integrals of Lipschitz-Hankel type involving products of Bessel functions, Phil. Trans. Roy. Soc. London, vol. A247, pp. 529551, April 1955. (references) [2] J. Clerk Maxwell, A Treatise on Electricity and Magnetism, 3rd ed., vol. 2. Oxford: Clarendon, 1892, pp.6873. [3] I. S. Jacobs and C. P. Bean, Fine particles, thin films and exchange anisotropy, in Magnetism, vol. III, G. T. Rado and H. Suhl, Eds. New York: Academic, 1963, pp. 271350. [4] K. Elissa, Title of paper if known, unpublished. [5] R. Nicole, Title of paper with only first word capitalized, J. Name Stand. Abbrev., in press. [6] Y. Yorozu, M. Hirano, K. Oka, and Y. Tagawa, Electron spectroscopy studies on magneto-optical media and plastic substrate interface, IEEE Transl. J. Magn. Japan, vol. 2, pp. 740741, August 1987 [Digests 9th Annual Conf. Magnetics Japan, p. 301, 1982]. [7] M. Young, The Technical Writer's Handbook. Mill Valley, CA: University Science, 1989.

Fig. 8. Bicubic interpolation out of the same scene.

Fig. 9. Super-Resolution image is clearer and with less noise as compare to bicubic interpolation (Figure 8) and Super-Resolution result can be further improve by increasing the number of frames taken into consideration. The top right edge of the paper are sharper compare to bicubic interpolation image.

VI. CONCLUSION In this paper, a simple Super-Resolution algorithm is implemented and presented. We see improvement in terms of the quality of the Super-Resolution over the original images taken from video camera. The algorithm implemented is fairly simple but fast as its able to compute Super-Resolution video stream in real-time. With this technique, one can take higer resolution picture with low resolution CCD.