2016 15th International Conference on Frontiers in Handwriting Recognition

Automatic Signature Segmentation Using Hyper-spectral Imaging

Umair Muneer Butt∗ , Sheraz Ahmad† , Faisal Shafait∗ , Christian Nansen‡ , Ajmal Saeed Mian§ , Muhammad Imran Malik∗†
∗ School of Electrical Engineering and Computer Science
National University of Sciences and Technology, Islamabad, Pakistan
Email:{14mscsubutt, faisal.shafait}@seecs.edu.pk
† German Research Center for Artificial Intelligence(DFKI), Kaiserslautern, Germany
Email:{firtname.lastname}@dfki.de
‡ University of California, Davis, USA
Zhejiang Academy of Agricultural Sciences, China
§ The University of Western Australia, Perth, Australia

Abstract— In this paper, we propose a method for automatic

signature segmentation using hyper-spectral imaging. The pro-
posed method first uses the connected component analysis
and local features to segment the printed text and signatures.
Secondly, it uses spectral response of text, signature, and
background to extract signature pixels. The proposed method
is robust, and remains unaffected by color and intensity of
the ink, and by any structural information of the text, as the
classification relies exclusively on the spectral response of the
document. The proposed method can extract signature pixels
either overlapping or non-overlapping from different back-
grounds like, logos, tables, stamps, and printed text. We used
high-resolution hyper-spectral imaging to study and classify
Fig. 1: Color Spectrum
300 documents with varying backgrounds. We evaluated the
proposed classification method and compared results with the
state-of-the art system. The proposed method outperformed the
state-of-the-art system and achieved 100% precision and 84%
recall. range is perceived by second type of cone and called as green
channel. Blue channel starts from 500 nm to 700 nm that
Keywords-Local Feature Analysis, SURF, Signature Segmen-
tation, Hyper-spectral Imaging is perceived by third type of cone. So in this way humans
perceive and visualize colors in RGB color space [1], [5].
Various research groups are working on both offline and
I. INTRODUCTION
online signature verification and identification systems [6],
Handwritten signatures are important biometric traits due [7], [8]. However these approaches are not fully realistic, as
to their social and legal acceptance for individual identifi- they assume that signatures are already segmented or they
cation and verification and to identify fraud in finance and do not overlap with other information like text, logos, and
banking industry [1], [2]. The widespread use of signatures noise etc., as shown in figure 2. In order to perform signature
in daily life justifies development of systems for automatic verification, it is required to first segment signatures from
signature segmentation and verification [3]. Handwritten the document. If segmentation process performed efficiently,
signature segmentation and verification is a difficult task it will improves the overall performance and accuracy of the
due to the free-flow nature of handwriting, and because verification process. As humans are trichromatic by nature
signatures some times are written on top of logos, lines etc. and machines are not trichromatic. Hyper-spectral imaging
Humans use three independent channels for color percep- (HSI) enables collecting and processing information across
tion [4], and that is the reason for use of trichromatic RGB the electromagnetic spectrum. HSI provide more fine and
color space in printers, cameras, and scanners. Similarly, detailed representation than RGB color space [9]. HSI has
RGB color space consists of three colors red, green, and application in different fields like astronomy, agriculture,
blue [1]. Figure 1 shows the spectrum of color which bio-medical imaging, geo-science, and mineralogy [10], [6],
contains the ultraviolet, visible, and infrared region. Humans [11].
can see things that lie only in the visible region. According Existing methods for signature segmentation use binary, gray
to RGB color space blue channel starts from 400 nm to 500 scale, or RGB images for segmentation [12], [13]. They
nm that is perceived by first type of cone. 450 nm to 630 nm use color or structural information of text to segment the

2167-6445/16 $31.00 © 2016 IEEE 19

DOI 10.1109/ICFHR.2016.14
 segmentation and enhancement of historic documents. Shiel
et al. [11] performed segmentation and text recovery using
HSI. They per formed segmentation using a 16th century old
book for segmentation whose cover was pasted down and
the late medieval texts which was written using multi-ink
combination. Aalderink et al. [24] propose a technique for
Fig. 2: Signature Scenarios in documents. (a) No Overlap restoring historic documents. They used spectral properties
(b) Partial Overlap (c) Complete Overlap of ink and extract the damaged areas of a 19th century book
by mapping ink particulars. Lettner et al. use multi-spectral
characteristics with AI techniques to enhance the visibility of
signature. However these methods are not effective as they historic documents [25] and perform segmentation on those
assumes that signatures are already segmented, signatures documents to save the text.
are not overlapping with other text, or they know the exact HSI is becoming increasingly popular due to algorithmic
position of signature which is unrealistic. advances like unmixing of spectra captured in a single image
We propose a system for automatic signature segmentation pixel [26], creating high resolution HSI Images with the aid
using hyper-spectral information and local features of the of high spatial density RGB images of the same scene [27]
text. The proposed system is the optimized version of the and methods for compressive sensing of HSI data [28].
state-of-the-art system presented by M. Malik et al. [14] with One of the key areas in document image analysis using HSI
100% precision and 73% recall. The recall of the state-of- is ink mismatch detection for detecting forgeries and fraud
the-art system is low as the band selection for printed and [29]. G. Reed et al. use HSI to differentiate between different
handwritten text is prone to error. In this paper, we improve inks and found it very useful for segmentation purpose.
upon the state-of-the-art by separating machine printed text They provide a detailed discussion about how different inks
and signature using the part-based features and connected red, green, blue, and black have different spectral response
component analysis [15], and then use hyper-spectral in- values which are useful for segmentation [30]. Z. Khan et
formation for extracting signature pixels. We evaluated our al. [31] use handwritten documents made using different ink
method on the same hyper-spectral signature dataset used by combination. They analyze the spectral response of inks in
the state-of-the-art system and achieved significantly better 33 bands and found it useful for detecting forgery with the
results then the state-of-the-art. assumption that forged part and genuine part contain same
The rest of this paper is organized as follows. Section II amount of text. They released a HSI dataset [32] publicly
provides an overview of the existing methods and the usage for ink mismatch detection and use k-mean clustering for
of HSI in different areas of research like document analysis, detecting forgery.
Medical, and agriculture etc. Section III gives details of the M. Malik et al. [14] use hyper-spectral information for seg-
data set we used in our experiments. Section IV provides menting the signature pixels. This is the first system reported
details of our automatic signature segmentation method. In on hyper-spectral signature data set. They handle both the
the section V results are presented. The paper is concluded overlapping and non-overlapping case of signature on text
with discussion and future work. where signature may complete overlap, partial overlap, or no
II. RELATED WORK overlap with the background, lines, logos etc. They extract
signature pixels with precision and recall of 100% and 73%.
Hyper-spectral imaging, also called optical reflectance
imaging, is a non-destructive method that calculates re- III. DATASET
flectance properties of a document with finer details and
In the last few years, many methods are presented for
high resolution [16], [17], [18]. In this section, we provide
signature segmentation from binary, gray scale, and RGB
an overview of techniques developed in the past few years
images [7], [33]. Up to the best of author’s knowledge,
along with the current research trends.
there are only two publicly available datasets, i.e., To-
Signatures are very important biometric trait due to their
bacco 800 [33] and Maryland Arabic [34] which can be
widespread use in person identification and verification in
used to evaluate the task of signature segmentation from
daily life [12], [19]. Financial institutions use signatures for
binary document images. The most recent and challenging
verifying persons identity in administrative and transaction
dataset is presented by M. Malik et al. [14], which consists
processes [20], [21].
of 300 invoices with handwritten signatures, captured using a
Over the last few years several efforts have been done
professional high-end hyper-spectral camera. Figure 3 shows
in the area of spectral imaging [22]. Easton et al. [13]
the hyper-spectral document capturing setup. The hyper-
present a novel and efficient method for forensic document
spectral camera used in the capturing setup covers spectral
comparison. A very efficient HSI system was developed by
region starting from ultraviolet (400 nm), visible, and ranges
Archives of Netherland for the analysis of historic docu-
to infrared region (900 nm) with a very fine resolution of 2.1
ments in libraries [23]. HSI is mostly used for text recovery,

20
 Fig. 3: Hyperspectral Imaging System

Table I: Hyper-spectral Imaging System Specification

Parameter Values
Fig. 4: Spectral Response Analysis
Specctral Range (nm) 400-900
Spectral Resolution (nm) 2.1
Spectral Channels 240
Spatial Channels 640 of signature, background, and printed text pixel across 240
Bit Depth 12
spectral bands. The response of printer ink and background
is almost consistent across all the bands. However, the
response of signature pixels varies a lot, especially in the
nm. Table I shows the complete specification of the hyper- bands near infrared region. This observation serves as a
spectral camera used to scan document images. The hyper- building block for our methodology.
spectral signature segmentation data set contains documents For training we used 30 document images from the dataset
with signatures performed by different authors on invoices. presented by Malik et al. [14] containing signatures and
The document images in dataset contain both overlapping machine printed text. To train the system, printed text
and non-overlapping signatures with rest of the content of is separated from signatures using available ground truth
the document i.e., lines, logos, and printed text. Signatures information, which results into two images against each
are done using different type of blue and black ink and gel document in training set i.e., printed text image and signature
pens. image. The following procedure is performed for signature
To evaluate the performance of signature segmentation meth- extraction:
ods, the data set is further divided into training and test 1) Connected components are extracted for each of the
set. The train set contains 30 real representative document printed and signature image present in the training set.
images with both overlapping and non-overlapping signa- 2) Compute the key points and their descriptors using
tures, while the test set contains remaining 270 images with SURF key point detector of all the connected com-
overlapping, non-overlapping, and no signatures. Patch level ponents of the printed text and added to machine
ground truth is provided for each image, where a bounding printed text (MPT) database. Similarly, key points and
box of signature regions is provided as a ground truth their descriptors are computed on all the connected
images. components of signatures and added them to hand
written signature (HRS) database as shown in figure 5.
Following steps are performed to segment signature from
IV. METHODOLOGY query image (from test set), which may contain printed text,
The proposed method for signature segmentation consists signature, and overlapped region.
of 3 phases. First, the document images are preprocessed 1) Connected components are extracted and for each
to remove noise. Second, connected component labeling connected component SURF features are extracted
and part-based features are used to automatically segment along with their descriptors. The descriptor of every
signatures from text. Finally, post processing is to performed point is compared with all the descriptors of printed
to precisely extract signature’s bounding box. text and signatures in the reference databases.
On analysis of the hyper–spectral response of document 2) K–Nearest Neighbour (K–NN) classifier is used to
images, it was seen that response of pen ink varies across classify connected components in on of two classes.
all of the 240 bands. Figure 4 shows the spectral response If a connected component key point has less distance

21
 filter. Preprocessed image is then used to find connected
components for separating the machine printed and signature
text. Once the blobs are located SURF key point detector
and descriptor is applied to get local information of the
text. The number of key points detected on each image
is referred as δ n , where n represents the band number as
shown in figure 6. Band where the amount of key points
are maximum is referred to as λmax and the band where
the amount of key points are minimum referred to as λmin .
Once these bands are located, the next step is to separate
the signature pixels from the remaining text or information.
To do this morphological opening is performed first on the
λmax that results in noise and signature pixels. The resulting
image is then subtracted from λmin called λsub so that only
the signature pixels remain. To get the signature pixels and
remove the noise, morphological closing is performed on
Fig. 5: Machine printed Text and Handwritten Signature λsub and the resulting pixels are used to extract the actual
Databases using Connected Component pixels from the document. In the post processing phase the
remaining noise and non text regions are removed using
based on geometric property so that only signature pixels
to MPT database descriptors as compared to HRS remain and noise like border lines, irrelevant dots, and
database descriptors, one vote is added to MP database Gaussian noise can be removed.
and vice versa. Repeat this process until all of the
V. EVALUATION
connected components are assigned to one of two
classes. As overlapping connected component contains For evaluation purpose we used precision and recall
both printed text and signature, it can not be classified measures. Precision tells us about the quality of the extracted
in one of the two classes based on K–NN distance. bounding box. In other words how many bounding box
3) To segment signature from overlapping region we that are extracted are actually signatures as shown in the
use hyper–spectral information. Based on the above equation 1. Second measure is recall which tells us how
observation, the first step is to locate the two most many signature bounding box our algorithm extracted out
significant and distinguishing bands automatically out of all signatures present in the document as shown in
of 240 bands which can be used for getting signature equation 2.
pixels. We have to locate the bands that have following Retrieved BBOX ∩ Signature BBOX
characteristics: Precision = (1)
Retrieved BBOX
• First band where all of the objects on the hy-
Retrieved BBOX ∩ Signature BBOX
per–spectral document have non–significant re- Recall = (2)
Signature BBOX
sponse including signatures named as λmax .
• Second band where everything is visible except Precision in the field of document analysis is the fraction
signatures named as λmin . of extracted signature bounding boxes that are relevant and
To find these two bands λmax and λmin we used local recall is the fraction of signature bounding box that are
key point detector. Local key point detectors locate most retrieved accurately.
important information in the documents which referred as As discussed in the previous section we have only patch level
key points. For the proposed methodology we used SURF as ground truth images which we used to find the precision and
key point detector and also as a descriptor. SURF key point recall. For every document ground truth image we have a
detector is a part based key point detector. As it extracts signature bounding box. We calculate precision and recall by
local information of the parts of the image which makes it assuming that if there is a 50% overlap between the extracted
more robust against the variations in the image. The pro- signature bounding box and ground truth bounding box are
posed method can also work with other key point detection considered as correct. Results are shown in table II.
methods such as; SIFT [33], FAST [35], or BRISK [36] etc. We extracted printed text and simple signature using Con-
Hyper–spectral document images consists of 240 gray scale nected Components and segment out the overlapped signa-
images each containing the spectral response of the docu- ture pixels using hyper-spectral information which improves
ment for that band and we apply local key point detector the system accuracy as compared to the state of the art
on each document. In the preprocessing phase noise sparks system. Firstly in the preprocessing phase we remove noise
are removed from the image using averaging and denoising which enhance the signature document image for finding

22
 Fig. 6: Signature Segmentation process

Table II: Segmentation Results realistic that it caters real time scenarios of overlapping and
non–overlapping text.
System Precision Recall
In the future we plan to improve ground truth from patch
Proposed Mehtod 100% 84% level to stroke level and increase the scope of our system by
M. Malik et al. [14] 100% 73% incorporating more realistic documents for segmenting and
also used for verification purpose.
ACKNOWLEDGMENT
connected blobs of printed text and signatures and then This research was partly funded by the ARC Grant
applied SURF features more intelligently by working only DP110102399, the UWA Grant 0060910300067, and Group
on the overlapped region which helps to improve the overall of Eight Australia–Germany joint research cooperation
performance. We use the same experimental setup with 30 scheme.
images as training set and 270 for testing and standard
R EFERENCES
performance measures. We improved the recall from 73% to
84% with 100% precision. Secondly in the post processing [1] R. Hunt, The Reproduction of Colour. The Wiley-IS&T Series
phase signature bounding boxes are formed more accurately in Imaging Science and Technology, Wiley, 2005.
by removing the background noise and pixels noise which [2] W. Song, S. Uchida, and M. Liwicki, “Comparative study
improves the bounding box quality. of part-based handwritten character recognition methods,” in
ICDAR, pp. 814–818, IEEE, 2011.
VI. F UTURE W ORK A ND C ONCLUSION
[3] S. J. Pinson and W. A. Barrett, “Connected component level
In this paper we present a method for automatically discrimination of handwritten and machine-printed text using
segmenting signature pixels from hyper-spectral document eigenfaces,” in ICDAR, pp. 1394–1398, IEEE, 2011.
images. We improve the state of the art system which is the
[4] S. Schneider, R. J. Murphy, and A. Melkumyan, “Evaluating
pioneer work in the signature segmentation field using HSI. the performance of a new classifier–the gp-oad: A compar-
Our method is the optimized version of the state-of-the-art ison with existing methods for classifying rock type and
system. We use Connected Component analysis and SURF mineralogy from hyperspectral imagery,” ISPRS Journal of
features with preprocessing and post processing steps, makes Photogrammetry and Remote Sensing, vol. 98, pp. 145–156,
the extracted signature bounding boxes more accurate and 2014.
efficient. Our method does not use any structural information [5] I. Ruyter, K. Nilner, and B. Möller, “Color stability of dental
of the text rather it uses the local features of the text and the composite resin materials for crown and bridge veneers,”
spectral reflectance produced by the object. As discussed in Dental Materials, vol. 3, no. 5, pp. 246–251, 1987.
section IV, pen ink, gel ink, printed text and background
[6] M. A. Calin, S. V. Parasca, D. Savastru, and D. Manea, “Hy-
have different spectral responses at different bands. Our perspectral imaging in the medical field: present and future,”
main task was to identify those distinguishing bands and use Applied Spectroscopy Reviews, vol. 49, no. 6, pp. 435–447,
them for segmentation. The proposed method is also more 2014.

23
 [7] R. Mandal, P. P. Roy, and U. Pal, “Signature segmentation [22] C. S. Silva, M. F. Pimentel, R. S. Honorato, C. Pasquini, J. M.
from machine printed documents using conditional random Prats-Montalban, and A. Ferrer, “Near infrared hyperspectral
field,” in ICDAR, pp. 1170–1174, IEEE, 2011. imaging for forensic analysis of document forgery,” Analyst,
vol. 139, pp. 5176–5184, 2014.
[8] J. Sita, B. Found, and D. K. Rogers, “Forensic handwriting
examiners’ expertise for signature comparison,” Journal of [23] R. Padoan, T. A. Steemers, M. Klein, B. Aalderink, and
Forensic Science, vol. 47, no. 5, pp. 1–8, 2002. G. De Bruin, “Quantitative hyperspectral imaging of historical
documents: technique and applications,” ART Proceedings,
[9] C.-I. Chang, Hyperspectral imaging: techniques for spectral 2008.
detection and classification, vol. 1. Springer Science&
Business Media, 2003. [24] B. Aalderink, M. Klein, R. Padoan, G. De Bruin, and
T. Steemers, “Clearing the image: A quantitative analysis of
[10] D. Lorente, N. Aleixos, J. Gómez-Sanchis, S. Cubero, O. L.
historical documents using hyperspectral measurements,” 37th
Garcı́a-Navarrete, and J. Blasco, “Recent advances and ap-
AIC, 2009.
plications of hyperspectral imaging for fruit and vegetable
quality assessment,” Food and Bioprocess Technology, vol. 5,
[25] M. Lettner and R. Sablatnig, “Spatial and spectral based
no. 4, pp. 1121–1142, 2012.
segmentation of text in multispectral images of ancient doc-
[11] P. Shiel, M. Rehbein, and J. Keating, “The ghost in the uments,” in 10th ICDAR, pp. 813–817, IEEE, 2009.
manuscript: Hyperspectral text recovery and segmentation,”
Codicology and Palaeography in the Digital Age, M. Rehbein [26] N. Akhtar, F. Shafait, and A. Mian, “Futuristic greedy
and PS und Torsten Schaßan, Eds. Norderstedt: Books on approach to sparse unmixing of hyperspectral data,” IEEE
Demand, pp. 159–174, 2009. Transactions on Geoscience and Remote Sensing, vol. 53,
no. 4, pp. 2157–2174, 2015.
[12] R. Plamondon and S. N. Srihari, “On-line and off-line hand-
writing recognition: A comprehensive survey,” IEEE Trans. [27] N. Akhtar, F. Shafait, and A. Mian, “Bayesian sparse repre-
Pattern Anal. Mach. Intell., vol. 22, no. 1, pp. 63–84, 2000. sentation for hyperspectral image super resolution,” in CVPR,
pp. 3631–3640, 2015.
[13] R. L. Easton, K. T. Knox, and W. A. Christens-Barry,
“Multispectral imaging of the archimedes palimpsest,” in [28] Z. Khan, F. Shafait, and A. Mian, “Joint group sparse pca
32nd Proceedings of Applied Imagery Pattern Recognition for compressed hyperspectral imaging,” IEEE Transactions
Workshop, pp. 111–116, IEEE, 2003. on Image Processing, vol. 24, no. 12, pp. 4934–4942, 2015.

[14] M. I. Malik, S. Ahmed, F. Shafait, A. S. Mian, C. Nansen, [29] L. J. Denes, P. Metes, and Y. Liu, Hyperspectral face
A. Dengel, and M. Liwicki, “Hyper-spectral analysis for database. The Robotics Institute, Carnegie Mellon University,
automatic signature extraction,” in 17th IGS, 2015. 2002.
[15] H. I. Koo and D. H. Kim, “Scene text detection via connected [30] G. Reed, K. Savage, D. Edwards, and N. N. Daeid, “Hy-
component clustering and nontext filtering,” IEEE Transac- perspectral imaging of gel pen inks: An emerging tool in
tions on Image Processing, vol. 22, no. 6, pp. 2296–2305, document analysis,” Science and Justice, vol. 54, no. 1,
2013. pp. 71–80, 2014.
[16] S. J. Kim, F. Deng, and M. S. Brown, “Visual enhancement of
[31] Z. Khan, F. Shafait, and A. S. Mian, “Towards automated
old documents with hyperspectral imaging,” Pattern Recog-
hyperspectral document image analysis,” in 2nd Int. Workshop
nition, vol. 44, no. 7, pp. 1461–1469, 2011.
on AFHA, Washington DC, USA, pp. 41–45, 2013.
[17] M. Golbabaee and P. Vandergheynst, “Hyperspectral image
compressed sensing via low-rank and joint-sparse matrix [32] Z. Khan, F. Shafait, and A. Mian, “Hyperspectral imaging for
recovery,” in ICASSP, pp. 2741–2744, IEEE, 2012. ink mismatch detection,” in 12th ICDAR, pp. 877–881, 2013.

[18] A. Chakrabarti and T. Zickler, “Statistics of real-world hy- [33] Y. Li, Y. Zheng, D. Doermann, and S. Jaeger, “Script-
perspectral images,” in CVPR, pp. 193–200, IEEE, 2011. independent text line segmentation in freestyle handwritten
documents,” IEEE Transactions on Pattern Analysis and
[19] M. Fairhurst, “Signature verification revisited: promoting Machine Intelligence, vol. 30, no. 8, pp. 1313–1329, 2008.
practical exploitation of biometric technology,” Electronics &
communication engineering journal, vol. 9, no. 6, pp. 273– [34] G. Zhu, Y. Zheng, D. Doermann, and S. Jaeger, “Multi-scale
280, 1997. structural saliency for signature detection,” in CVPR, pp. 1–8,
IEEE, 2007.
[20] M. C. Fairhurst and E. Kaplani, “Perceptual analysis of
handwritten signatures for biometric authentication,” IEE [35] E. Rosten and T. Drummond, “Fusing points and lines for
Proceedings - Vision, Image and Signal Processing, vol. 150, high performance tracking,” in 10th ICCV, vol. 2, pp. 1508–
no. 6, pp. 389–394, 2003. 1515, IEEE, 2005.
[21] F. Leclerc and R. Plamondon, “Automatic signature verifica- [36] A. Alahi, R. Ortiz, and P. Vandergheynst, “Freak: Fast retina
tion: the state of the art 1989-1993,” International Journal of keypoint,” in CVPR, pp. 510–517, IEEE, 2012.
Pattern Recognition and Artificial Intelligence, vol. 8, no. 3,
pp. 643–660, 1994.

24