910 IEEE TRANSACTIONS ON PATTERN ANALYSIS A N D MACHINE INTELLIGENCE, VOL. IO. NO.

6, NOVEMBER 1988

A Robust Algorithm for Text String Separation from

Mixed Text/Graphics Images
LLOYD ALAN FLETCHER AND RANGACHAR KASTURI, MEMBER, IEEE

Abstruct-An automated system for document analysis is extremely The initial processing stage in an automated document
desirable. A digitized image consisting of a mixture of text and graph- analysis system requires conversion of paper-based
ics should be segmented in order to represent more efficiently both the
areas of text and graphics. This paper describes the development and
graphics/text to a digital bit-map representation. Wher-
implementation of a new algorithm for automated text string separa- ever the primary goal of the automated document analysis
tion which is relatively independent of changes in text font style and system is interpretation of graphic data, text strings pres-
size, and of string orientation. The algorithm does not explicitly rec- ent within the digitized document must first be separated
ognize individual characters. The principal components of the algo- from the graphics in order that subsequent processing
rithm are the generation of connected components and the application
of the Hough transform in order to group together components into
stages may operate exclusively on the graphic informa-
logical character strings which may then be separated from the graph- tion. The extracted text may be stored separately for input
ics. The algorithm outputs two images, one containing text strings, and to a character recognition system for later retrieval or re-
the other graphics. These images may then be processed by suitable vision. Since document types vary widely in style and
character recognition and graphics recognition systems. The perfor- content of both graphic and text data, an algorithm to per-
mance of the algorithm, both in terms of its effectiveness and compu-
tational efficiency, was evaluated using several test images. The results
form text string removal must be able to accommodate
of the evaluations are described. The superior performance of this al- documents containing text of various font styles and sizes.
gorithm compared to other techniques is clear from the evaluations. Further, the documents may contain text strings which are
Zndex Terms-Connected component analysis, document analysis, intermingled with graphics, and text characters which are
image processing, image segmentation, image understanding, text and similar in size or shape to graphics. In general, text strings
graphics separation, text recognition. may be of any orientation in the image; not simply hori-
zontal or vertical but possibly diagonally aligned.
INTRODUCTION Several algorithms for text string separation have been
reported in the literature [6]-[9]. However, many of these
T HE move from paper-based documentation towards
computerized storage and retrieval systems has been
prompted by the many advantages io be gained from the
algorithms are very restrictive in the type of documents
they can process and are therefore not useful in a general
electronic document environment. Document update automated document analysis system. For example, the
and revision is efficiently achieved in the computerized combined symbol matching algorithm [7] is sensitive to
form. For efficient processing and storagz of documents, changes in text font style and size: the algorithm must be
however, it is necessary to generate a description of run once for each font type using different parameters,
graphical elements in the document rather than a bit-map words of less than three characters which are embedded
in order to decrease the storage and processing time. Thus, in longer strings are not removed, and the string removal
increasing emphasis is being placed on the need for the is only along a specified orientation. The Block Segmen-
realization of computer-based systems which are capable tation technique [8] broadly classifies regions into text or
of providing automated analysis and interpretation of pa- graphics; i.e., characters which lie within predominantly
per-based documents [ l]-[3]. Much of the attention paid graphics regions are classified as graphics. The Bley al-
to automated document analysis systems in the literature gorithm [9] is also sensitive to variations in text font style
has been in relation to engineering drawings and diagrams and size; the algorithm breaks connected components into
[4], [5]. Such systems provide a means for originating subcomponents, which makes it difficult to process the
technical information (text and graphics) in a digital form components for graphics recognition. Thus there is no
suitable for interactive graphics editing, reproduction, and single algorithm which is robust enough to segment im-
distribution. ages containing mixed graphics and text, with multiple
font styles and sizes and strings of arbitrary orientation.
Manuscript received August 8, 1986; revised June 12, 1987. Recom-
mended for acceptance by J. Kittler. This work was supported in part by This paper describes a new robust algorithm for text string
the National Science Foundation under Grant ECS-8307445, separation from mixed text/graphics images.
L. A. Fletcher was with the Department of Electrical Engineering, the
Pennsylvania State University, University Park, PA 16802. He is now with
Bell Communications Research, 331 Newman Springs Road, Red Bank, A ROBUSTALGORITHM FOR TEXTSTRING SEPARATION
NJ 07701.
R. Kasturi is with the Department of Electrical Engineering, the Penn-
A robust algorithm for text separation has been de-
sylvania State University, University Park, PA 16802. signed to separate text strings from graphics, regardless
IEEE Log Number 8823861. of string orientation and font size or style. The algorithm

0162-8828/88/1100-0910$01.OO 0 1988 IEEE

FLETCHER A N D KASTURI: TEXT STRING SEPARATION 911

VOLTAGE REGULATORS AND POWER CIRCUITS 4

uses simple heuristics based on the characteristics of text 184 Heat and power design m
0
strings. The algorithm performs no character recognition rn

and can only separate characters from text on the basis of * * %

size and orientation. Thus, for proper operation, the doc-

ument should meet the following requirements:
1) The range in text font sizes within an image should
be such that the average height of the characters belong-
ing to the largest font is not greater than five times the
average height of characters belonging to the smallest
character font.
* *
* *
2 ) The interline spacing between parallel lines of char- * *
acters should not be less than one quarter of the average * *
height of characters in the line having the larger font size. * *
3) The resolution of the digitizer used should be such 16384Hi

that each text character forms an isolated component not 4n1

connected to graphics or other characters.

4) The largest character in any phrase should not have
an area greater than five times the average area of all char-
acters in the phrase.
5 ) The intercharacter gap between characters in the
same word should not be greater than the average local
character height. Also, the interword gap between words
of the same phrase should not be greater than 2.5 times ing rectangles. Each rectangle thus identifies a single con-
the average local character height. nected component. The output of the connected
If these constraints are satisfied for any particular im- component generation algorithm is an information array
age, the performance of the algorithm will be highly re- which specifies: the maximum and minimum coordinates
liable. However, acceptable segmentation is obtained of the circumscribing rectangles of connected compo-
even if some of these constraints are not strictly satisfied. nents, the coordinates of the top and bottom seeds of each
For example, the algorithm will correctly segment text connected component, and the number of black pixels.
strings even when some of the characters are eight con- Black pixel density and dimensional ratios are easily ob-
nected to each other, as long as there are at least three tained from this information. This algorithm is described
components in each string. in detail in [101. Although this algorithm is similar to the
The algorithm described here is designed to accomplish algorithms described in the literature [ l l ] , [12], in our
the following without explicitly recognizing individual algorithm we do not change the value of each pixel to its
characters : corresponding component label, but retain the coordinates
Locate potential text strings and separate them from of top and bottom seeds for each component.' Each con-
the large graphics. nected component is then either rejected or accepted as a
Examine the strings and separate all phrases that con- member of a text string based on its attributes (size, black
tain at least three characters. pixel density, ratio of dimensions, area, position within
Generate two images, one each for the text strings the image etc.).
and graphics. The enclosing rectangles corresponding to the con-
The algorithm consists of the following steps: nected components of Fig. 1 are shown in Fig. 2. Note
Connected component generation. that the largest graphic is enclosed by a single rectangle.
Area/ratio filter. Also, the long line near the top of the image is enclosed
Collinear component grouping. by a rectangle. This rectangle has a height which is sig-
Logical grouping of strings into words and phrases. nificantly greater than the thickness of the line since the
Text string separation. line is oriented at a slight angle to the image axis.
These processing steps are describei in detail in the fol-
lowing sections. A test image of 2048 x 2048 pixels, AreajRatio Filter
shown in Fig. 1 , is used to demonstrate the performance By initial examination of connected component attri-
of this algorithm. butes, the working set of connected components can be
reduced to one which contains a higher percentage of
Connected Component Generation
'An image typically contains more than 256 components. Labeling each
Connected component generation involves grouping to- pixel with its corresponding component number would require at least 2
gether black pixels which are eight connected to one an- bytes per pixel. This would double the memory requirement to 8 MB for a
other (assuming a black image on white background). In 2048 X 2048 pixel image and significantly deteriorates system perfor-
mance (through increased page faults). However, this necessitates multiple
this technique, eight connected pixels belonging to indi- passes in the text string removal stage to remove all pixels belonging to a
vidual characters or graphics are enclosed in circumscrib- particular connected component.
912 IEEE TRANSACTIONS ON PATTERN ANALYSIS A N D MACHINE INTELLIGENCE, VOL. IO. NO. 6, NOVEMBER 1988

than this threshold. Since text characters of the same size

r I
may have different areas, the histogram is blurred such
o n U that neighboring areas are grouped together in order to
0 0 determine the most populated area.
O D
A similar type of filtering approach is taken with the
o n
0 0 connected component dimensional ratio attribute. Very
n o long lines are unlikely to be text characters, and should
0 0 therefore be discarded. Isolated straight lines within the
o n image may be discarded on the basis of dimensional ratio.
o n ULb2n
If a connected component has a ratio of less than 1 :20 or
o n
o n
Qm
r greater than 20 : 1 , it is discarded.
O D
0 0
~ I
Collinear Component Grouping
Text strings are composed of characters which are ori-
ented along the same straight line. In order to logically
connect or group together characters into strings, it is nec-
essary to determine which characters in the image actually
~ m r n r n c a m n ~ m l m

- 0 0 - =+#==?- I
0
lie along any given straight line. By grouping components
into strings associated with a particular line, the compo-
Fig. 2 . Rectangles enclosing the connected components of test image 1. nents can be ordered according to their distance along the
line. Further grouping may then take place by examining
characters. In general, a mixed text/graphics image will the distance between characters. By comparing the inter-
produce connected components of widely varying areas. character distance with the interword gap and interchar-
The larger connected components represent the larger acter gap thresholds, the string can be segmented into log-
graphic components of the image. It is desirable to locate ical character groups, that is, into logical words and
and discard large graphics in order to restrict processing phrases. Only if components belong to a logical character
to components which are candidates for members of text group can they be regarded as members of a valid text
strings, thereby improving both accuracy and processing character string. Such components are identified and are
speed. separated from the image. These steps are explained in
By obtaining a histogram of the relative frequency of detail in this and the following sections.
occurrence of components as a function of their area, it is The Hough transform is a line to point transformation
possible to set an area threshold which broadly separates [ 131 which, when applied to the centroids of connected
the larger graphics from the text components. By correct components in an image, can be used to detect sets of
threshold selection, the largest of the graphics can be dis- connected components that lie along a given straight line.
carded, leaving only the smaller graphics and text com- A line in the Cartesian space (x,y ) given by the equation
ponents as members of the working set of connected com-
ponents. The area threshold selection procedure must
p = x cos 6 + y sin 8 (1)
ensure that the threshold lies outside that part of the his- is represented by a point ( p , 0 ) in the Hough domain.
togram which broadly represents the set of text charac- Similarly, every point (x,y ) in the Cartesian space maps
ters. The threshold itself is determined using the area his- into a curve in the Hough domain. Thus to locate all the
togram (as opposed to presetting it to a value) although a connected components that are collinear, the Hough
priori knowledge about the types of documents being pro- transform is applied to the centroids of the rectangles en-
cessed (such as the amount of text data compared to closing each connected component. All the curves corre-
graphics), if available, is helpful in determining this sponding to the collinear components intersect at the same
threshold. For documents that contain a substantial point ( p , e ) , where p and 8 specify the parameters of the
amount of text and some large graphics, the most popu- line.
lated area will, in general, represent mostly text compo- In the discrete case, the Hough domain is a two-dimen-
nents or, at least, small graphics. By ensuring that the sional array representing discrete values of p and 8. The
threshold is set above the most populated area, A,, the resolution along the 8 direction is set to one degree. The
possibility of discarding members of the text character set resolution along the p direction is treated as a variable R .
is avoided. This alone is not adequate to process images Optimal selection of R is critical to correct grouping of
that contain text strings of different sizes (it is likely that connected components. This is because in any text string,
the most populated area corresponds to the smallest sized the centroids of all characters belonging to a phrase are
characters). Thus, a second parameter, the average area not necessarily collinear. For any phrase, character
Aavg, is computed. The area threshold is then set at five heights vary (upper, lower case) and character positions
times the larger of the two parameters A , and Aavg.The vary in relation to each other (ascenders, descenders). In
histogram is searched to locate components that are larger general, the centroids of characters belonging to the same
FLETCHER AND KASTURI: TEXT STRING SEPARATION 913

phrase will lie in the range k6,. from the axis of the line 2) Set the Hough domain resolution R to 0.2 X H,,T.
connecting the phrase members. However, 6, is a function Set a counter to zero.
of the text string height. Thus, it is important to select a 3) Apply the Hough transform to all components in the
Hough domain resolution, before applying the transform, working set for 8 in the range: 0" I8 I5 " , 85" I0 I
which will cause the majority of component centroids be- 95", 175 5 0 I 180". (In our implementation p is al-
longing to a phrase to be grouped into the same cell. The lowed to have negative values and Om,, = 180.) Set the
optimal resolution is dependent on the heights and relative running threshold RT, = 20.
positions of characters within a string. Too large a value 4) For each cell having a count greater than RT,., per-
for R may cause several parallel strings to be grouped into form steps 5-10.
a single cell (over-grouping). If R is too small then con- 5 ) Form a cluster of 11 p cells (constant e), including
nected components belonging to the same string may be the primary cell, centered around the primary cell.
grouped into different cells (undergrouping). The value of 6 ) Compute the average height of components in the
R should depend on the average local character height, cluster Ha.
Ha, for a line 1. However, since Ha cannot be determined 7) Compute the new clustering factor fclus = H , / R .
until characters have actually been detected as belonging 8 ) Re-cluster f fclus cells centered around the primary
to a collinear string, and since R must be known before cell, including the primary cell.
such grouping, an initial estimate for R is made based on 9) Perform string segmentation (explained in detail in
the average height, H,,, of all connected components in a later section).
the current working set. The resolution is then set to: R 10) Update the Hough transform by deleting the con-
= 0.2 x If%,$. tributions from all components discarded in step 9.
The resolution R allows for some perturbations in the 11) Decrement RT, by one. If RT, is greater than 2, go
collinearity of components belonging to text strings. to step 4.
However, this will not be adequate to group all ascenders 12) If the counter is equal to 1 then stop. Otherwise
and descenders in a character string. Further, the value of compute the Hough transform for the remaining compo-
R , which is based on average character height, will not be nents for 8 in the range: 0" I8 < 180". Reset RT, to
optimal for grouping text strings of different heights. 20. Increment the counter by one. Go to step 4.
Thus, clustering in the Hough domain may become nec- Step 10 above is performed in order to "refresh" the
essary for each primary cell: Here, "primary cell" refers Hough domain. If the Hough array is not continually up-
to the cell containing the majority of the connected com- dated, discarded characters contribute to cell counts de-
ponents in a string. In order to group together all com- spite having been removed. This can lead to a severe deg-
ponents belonging to a string, neighboring cells must be radation in processing speed since information about
clustered into the primary string. The required degree of discarded components is unnecessarily extracted from the
clustering, ticlus,
will depend on the local character height. Hough domain. Thus, the refreshing of the Hough do-
However, when strings are initially extracted, I1 cells main is designed to improve performance.
centered around the primary cell are grouped for exami-
nation. Once the string has been extracted, the local clus- Logical Grouping of Strings into Words and Phrases
tering factor can be determined. Fig. 3 shows the connected components associated with
The average local character height for a string Ha is the line lpr. Note that the centroids of these components
used to determine the degree of clustering around the pri- do not necessarily lie on the line Zpr. It is necessary to
mary cell. This will ensure that all ascenders and de- order these components according to their distance along
scenders in a string are included in the text string group- the line [from a reference point, say ( 0 ,y o ) ] . That is, for
ing. The clustering factor is then set to H a / R . each component the distance along the line lpr is calcu-
Since the vast majority of documents contain text strings lated.
which are parallel to the document axes, it is advanta- In order to locate words and phrases, the positional re-
geous to remove these strings first. To allow for orienta- lationships between connected components are examined.
tion errors during digitization, a tolerance of five degrees It is the intercharacter gap and interword gap thresholds,
is allowed in processing horizontal and vertical lines. T, and T,, which are used to determine component group-
Strings at other orientations are processed after consider- ing. These parameters are functions of the character
ing the horizontal and vertical strings. Furthermore, more height. In computing this, an average of the height of sev-
populated cells in the Hough domain are processed before eral local characters should be used rather than simply
processing less populated cells. This order of string pro- using the height of the component currently under exam-
cessing gives greatest weight to longer strings. If shorter ination. For example, the component under examination
phrases or words were treated before longer ones, a deg- may be a period at the end of a string, which would have
radation in performance would be observed. very small height. For this reason the four neighboring
In summary, extraction of strings from the Hough do- (two on each side) characters are used, along with the cur-
main is performed as follows: rent component, to determine the thresholds T, and T,,.
1) Calculate the average height of components H,, in These operations are explained in the following para-
the working set. graphs.
~

914 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 10, NO. 6, NOVEMBER 1988

Y Type field may have the label 3 , w, or p (ini-

tialized to i), specifying the group type. The fields
Head and Tail serve as pointers to the first and last con-
nected components within the group. The last field Num
contains the number of connected components belonging
to the group. The data structure Phrases is identical to
Groups except that it does not possess the Type field.
2) For each connected component associated with the
currently active string, determine T, and T, based on the
neighborhood average character height H,. If String;( D e )
IT, then elements i and i + 1 belong to the same word
group. If the current group is a new one, set
Groupsgn(head ) = i and Groupsgn( t a i l ) = i + 1, where
gn is the number of groups currently detected in the string.
Fig. 3 . Collinear component ordering. Otherwise set Groupsgn( t a i l ) = i + 1. Set
Groupsgn(num) = Groupsgn(num ) + 1. If the group type
is not p, set Groups,,(type) = w. Continue the T,
Associated points of the line I, once ordered, become threshold test until the current group is terminated. If the
part of a data structure containing all information about T, test fails, terminate the group entry by advancing the
components belonging to all collinear strings. An entry is value of gn by one.
made in the string information array for each associated 3 ) Compare String,(D,) with T,. If De < T, then the
point, pointing to the connected component to which each previous group and the current connected component be-
point belongs. The information contained in the con- long to the same phrase. Make entries in Phrases to point
nected component array is then used to compute the edge- to the member groups. Set Groups,,(type) = p and
to-edge distance, De, for every pair of adjacent compo- Groupsgn- , ( t y p e ) = p. Begin a new group entry for
nents. The magnitude of De belonging to the associated the associated connected component i. Repeat step 2. If
point i, is the distance between the two closest edges of the T, threshold is not satisfied then the adjacent con-
the connected components associated with collinear com- nected components currently under consideration do not
ponent elements i and i + 1. belong to the same phrase: terminate any active entry in
By placing a window over each connected component Phrases. Begin a new group by repeating step 2.
associated with the string information array, and consid- 4) Once all entries in String have been classified, search
ering four nearest neighbors, the average height of con- for two or more consecutive i groups present within a
nected components within the window H, is determined. phrase. There should never be more than two consecutive
In computing this average the vertical dimension of the isolated characters embedded within the same phrase. Re-
enclosing rectangles is used for lines oriented between 0 move any such sequences by splitting or truncating groups
and 45 degrees and the horizontal dimension is used for as necessary.
strings oriented at angles greater than 45 degrees. T, is The results of the string segmentation are two data
the intercharacter gap threshold, set at T, = H,. The in- structures which provide information about words belong-
terword gap threshold T, is set at 2.5 X Ha. Two adjacent ing to a particular phrase and information about the con-
connected components separated by an edge-to-edge dis- nected components which make up any words within a
tance De of less than or equal to T, belong to the same phrase.
word group. Word groups separated by no more than T, The problem of destructive line overlap occurs if
belong to the same phrase group. Connected components shorter strings, which overlap longer strings, are pro-
which belong to neither of these types are labeled as iso- cessed before the longer strings. For example, consider
lated characters. Note, however, that single characters the strings shown in Fig. 4. All the components in Fig.
which are embedded in a multiword phrase become part 4(a) belong to at least one phrase. Thus, all of these com-
of the phrase group. Isolated words of less than N ( N = ponents must be removed. However, if short strings are
3 in our experiment ) characters are not removed from the removed first, we get Fig. 4(b). Finally, after considering
image. Further, when several isolated characters satisfy the longer strings, we get Fig. 4(c). Thus, it is important
the interword gap to form a phrase containing more than to consider longer strings before shorter strings for group-
three characters, the condition is checked and the char- ing and removal. However, these strings belong to differ-
acters are not removed. ent collinear component clusters in the Hough domain.
The string segmentation algorithm is summarized as Therefore, it is necessary to examine all clusters to sep-
follows: arate longer strings before removing shorter strings.
1) Define the data structure Srring of which each ele- The string processing procedure thus gives weight to a
ment is an associated point belonging to the same line. phrase group depending on the number of connected com-
Define the data structure Groups, a vector of 100 ele- ponents belonging to the particular group. In examining
ments. Each element of Groups contains four fields. The cells for the extraction of collinear component sets, the
FLETCHER AND KASTURI: TEXT STRING SEPARATION 915

process described earlier, runs of eight connected black

pixels are merged together. If a black pixel belongs to a
group to which one of the seeds also belongs then the
pixel is marked for deletion. The scanning procedure is
repeated four times, twice for each seed. The procedure
is begun with the top seed, moving south; then starting
from the bottom seed and scanning north; then east from
the top seed again; finally the component is scanned west
from the bottom seed. This multidirectional scan ensures
that all possible eight connected groups are detected. All
black pixels marked during this procedure are then re-
placed with white pixels.
Do [10 The algorithm is implemented in Pascal. The program
(c) was run on a DEC VAX 11/785 running the VMS oper-
Fig. 4 . Effect of processing shorter strings before longer strings: (a) char- ating system. The complete program includes over 1000
acters oriented along three lines; (b) result of processing shorter strings lines of code. The following section presents the results
first; (c) final result: some of the characters are not removed. obtained by application of the algorithm to several test
images.

running threshold RT, is used. The running threshold is EXPERIMENTAL RESULTS

also used as the threshold for the phrase population. Char- In order to test and evaluate the performance of the al-
acter groups are only processed if the group population gorithm, several test images were required. Several doc-
exceeds both RT, and N . This avoids the confusion which uments were digitized to 2048 x 2048 pixels using a Mi-
can result from destructive line overlap. Further refine- crotek Image Scanner (interfaced to an IBM PC-AT) with
ments are also applied to discard disproportionately larger a resolution of 300 pixels per inch. The scanned images
connected components, possibly due to graphics, from in compressed form were transfered to the VAX using
text strings. KERMIT communications software. The results of the al-
Words or phrases which consist of several repeated gorithm application to each image are illustrated and dis-
characters, such as dotted or dashed lines, should not be cussed in [lo]. In this paper we limit the discussion to
removed from the image. Such components are in fact part two test images.
of the graphics rather than actual text strings. Groups of The image shown in Fig. 1 has several characteristics
such repeated characters possess a high degree of con- found in different types of mixed text/graphics images:
sistency in both ratio and black pixel density. Hence, re- various text font styles and sizes; text strings enclosed by
peated strings can be detected and discarded by calculat- graphics; text strings with various orientations; graphics
ing the variance in ratio and density for each phrase of a of various size and shape. In general, most document
collinear component set. If both the variance in ratio and types do not possess such a wide variation in text font
density fall below a given threshold for three or more con- styles, sizes, and orientations within the same document.
secutive characters, then these repeated characters of the However, if the algorithm could be made robust to such
phrase group are discarded from the working set and the variations in image characteristics, then it would also be
current collinear component set, and are not deleted from robust when applied to more uniform images.
the image. The result of the removal of all phrases of three or more
characters is shown in Fig. 5. Note that several of the
Text String Separation characters in the diagonally oriented table, such as g
Once all strings have been extracted from the Hough and p are not removed. These letters are in fact eight
transform, all connected components corresponding to connected to the table itself, as can be seen from the con-
these strings are deleted from the image array. This in- nected components in Fig. 4. Thus, these characters are
volves replacing black pixels, belonging to marked con- part of the graphics and therefore cannot be isolated for
nected components, with white pixels. It is important that identification as characters. Note that the series of aster-
only those black pixels which originally formed a partic- isks have not been removed since they constitute repeated
ular connected component should be deleted, not simply isolated character phrases. Although the inter-com-
all black pixels which lie within the area of the circum- ponent gap satisfies the interword gap threshold, it does
scribing rectangle. It is for this reason that the seeds of not satisfy the inter-character gap threshold. As a result,
the connected components are retained during connected the string is discarded since it is a phrase made up of sin-
component generation. Knowing the location of Seed, and gle character words. Of the three dashed lines in the bot-
Seedb of the connected component allows all black pixels tom right of the image, two strings have been partially
which are eight connected to them to be detected and removed. The ratio consistency of these strings was not
marked for deletion. Using a line by line scan method sufficient to prevent their removal as text. The consis-
similar to that used in the connected component formation tency in both ratio and density of the other dashed line
916 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 10, NO. 6, NOVEMBER 1988

* *
* * r - 1

* *
* *
* *
* *
* *
* *
* *
* *
* *
* *
* *

A
I
I Fig. 6. Test image 2 .
I
/* I
- I

Fig. 5 . Output after text string separation.

was high enough for it to be retained as a graphic. Note

that three of the small graphics in the neighborhood of the
battery figure have been deleted. These were grouped to-
gether as a three component phrase since the inter-word
gap threshold was satisfied.
The image of Fig. 6 has a large number of text strings.
In this respect this image complements the first test im-
age. Due to ink spread in the original document several
characters are connected to each other. For example, the
characters s and e in most of the occurrence of the word
Base are connected to each other as can be easily ver-
ified in the connected components diagram shown in Fig. Fig. 7. Connected components of test image 2.
7. In several instances the word contains only two com-
ponents. The output image after text extraction is shown 1 I 10 11

in Fig. 8. All character strings, including those that are mo

Si
eight connected to each other, have been removed with
one notable exception: the letters S and k in the
work Stack in second column. In this word the letters S .

a and c are connected to each other making it a string of

four characters. The two connected components in the
middle of the string are removed as parts of vertical strings SI

(longer than 4 characters) before the algorithm looks for DI

strings which contain four characters. Thus the letters S

and k are left behind. The reasons for other components
BX
which are not removed are as follows: two of the 0s in
the string 000 are connected to each other making it a Fig. 8. Output after text string separation.
two character string: the intercharacter gap between the
string SI at the top of the second column and its pre- strates the robustness of the algorithm to images of var-
vious character is large compared to the height of the pre- ious types. In this section the algorithm is evaluated in
vious character (the + sign is made up of two disjoint terms of processing speed.
components of relatively small heights; see Fig. 7). Sim- The time required to generate connected components
ilar comments hold good for the strings 10 and 11 that are depends not only on the number of components within the
next to = signs. Other strings which have not been image, but also on the complexity of the information rep-
removed contain less than three characters. resented by the black pixels. In general, however, pro-
cessing time increases in direct proportion to the number
Per$ormance Evaluation of connected components in the image. The time required
The results of the application of the text string separa- ~~
for the application of the Hough transform to the active
tion algorithm, given in the previous section, demon- set of components also increases with the number of con-
FLETCHER A N D KASTURI: TEXT STRING SEPARATION 917

nected components. However, the time for a single Hough TABLE I

transform consumes only about five percent of the total BREAKDOWNOF CPU TIMESFOR PROCESSING
TESTIMAGE 1 A N D FOR
PROCESSING
STEPSFOR A COLLINEAR
COMPONENTSETOF POPULATION 15
processing time for an entire image.
Processingstep CPU time (sea.) CPU lime (sen.)
The algorithm is implemented in several distinct steps. for each operation for whole image
In Table I, a breakdown of the processing times required
for several operations applied to Test Image 1 is given. 1. Read 2 K image 48.5
2. Write 2K image to disk 50.8
Times for operations on both the whole image (reading,
3. Generation of CCs (395) 47.7
writing, connected component generation and deletion) 4. Initialize Info array 0.14
and operations on character strings (segmentation, area 5. Filterout Ccs based
thresholding) are given. Steps 1-7 and step 12 in the table on area and ratio 0.47
6. Hough mslonn
are operations performed on the whole image array. Steps (393 ccs.0. 90) 9.3
8-1 1 are performed on a single collinear component set. 7. Hough transform
(270 CCs. all angles) 18.9
The CPU time for each of these operations is given for a 8. Clustering and Swing
collinear component set of 15. exfraction 0.41 243.3
9. Sning element initialization 0.04 94.8
Several string operations typically require processing
IO. String grouping 0.42 148.5
times of only a few tenths of a second. These processing 11. String refinement 0.11 94.3
times vary with the number of connected components in 12. Connected component deleiion 8.50
a collinear component set. Although individually the Total 764.31
string processing steps require relatively small processing
time, when several hundred collinear component sets are
processed (as in a typical image) the cumulative process- on local component height). Such modifications will r e
ing time becomes significant and is directly related to the sult in a degradation in processing speed. There is a def-
number of components in an image and the complexity of inite tradeoff between the robustness of the algorithm and
the image. The processing time increases as the cell pop- the processing speed. Making the algorithm more adapt-
ulation threshold decreases, since more cells become able will result in longer processing times.
available for extraction. Many more collinear component Some problems are encountered when dealing with dot-
sets of lower population are processed than sets of higher ted or dashed lines. The consistency in density and ratio
population. As the population threshold RT, decreases, in such graphics strings is not always as high as might be
the processing time for the working set rapidly increases. expected. Also, the intercharacter gap is not always con-
sistent, resulting in the classification of some dashed lines
Algorithm Improvements as character strings. Some further processing may be re-
The algorithm is highly CPU intensive. In order to be quired in order to fully accommodate the detection of re-
useful in automated document processing, the processing peated graphic strings.
time should be minimized. A hardware implementation of SUMMARY AND CONCLUSIONS
the algorithm would achieve a reduction in the required
A new algorithm for text string separation from mixed
CPU time. However, certain modifications might be made
text/graphics images has been presented. The algorithm
to the algorithm itself to improve performance. For ex-
is robust to changes in text font style and size within an
ample, collinear component set extraction from the Hough
image. The algorithm also accommodates the separation
domain involves, effectively, reapplication of the Hough
of text strings of any orientation. The algorithm adapts to
transform to all active components for a given cell with
changes in text characteristics within the image in order
values of p and 8. The required processing time for these
to achieve optimal performance. The algorithm is highly
steps could be reduced drastically if each Hough domain
reliable provided that documents conform to several con-
cell contained information about the components which
straints on their characteristics. The algorithm presented
belong to it. This would require cells to maintain a list of
has several advantages over previously applied tech-
possibly up to one hundred component entries. Due to the
niques. With improvements made to the algorithm in
limited memory on our computer, it was not possible to
terms of processing speed and efficiency of data represen-
implement this feature.2
tation, the application of the algorithm will become highly
It is possible that the algorithm may be made even more
appropriate for use in document analysis systems.
robust to rapid changes in image characteristics over small
areas of the image. Several of the processing steps can be REFERENCES
made more adaptive by using several iterations in order [ l ] K. Y. Wong, R. G. Casey, and F. M. Wahl, Document analysis
to gather more accurate statistics about strings within the system, IBM J . Res. Develop., vol. 6 , pp. 642-656, Nov. 1982.
image (e.g., calculation of the thresholds T, and T, based [2] R. N. Slater, Automating data base capture for CAD/CAM, Com-
put. Graphics World, vol. 48, pp. 45-53, Oct. 1984.
[3] M. Karima, K. S . Sadhal, and T. 0. McNeil, From paper drawings
*Our system has 8 MB of memory. Executing a program containing more to computer aided design, IEEE Comput. Graphics Applications,
than one 2048 x 2048 byte variable array resulted in significant deterio- vol. 5 , pp. 24-39, Feb. 1985.
ration in performance of the system due to increased page faults. With a [4] C. L. Huang and J. T. Tou, Knowledge based functional symbol
system having more memory, larger data structurescould be used to greatly understanding in electronic circuit diagram interpretation, Aplica-
enhance performance. tions ofArtifcia1 Intell. III, Proc. SPIE, vol. 635, pp. 288-299, 1986.
918 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. IO, NO. 6, NOVEMBER 1988

[ 5 ] H. Bunke, Automatic interpretation of lines and text in circuit dia- of Technical Staff with Bell Communications Research, Red Bank, NJ. He
grams, in Pattern Recognition Theory and Applications, J . Kittler, is currently working in the Switching Analysis and Reliability Technology
K. S . Fu, and L. F. Pau Eds. Boston, MA: D. Reidel, 1982, pp. Center, where he is involved with the development of criteria to assure the
297-3 10. reliability and quality of digital switching systems and telecommunications
[6] L. T . Watson, K. Arvind, A. W. Ehrich, and R. M. Haralick, Ex- equipment deployed by the regional Bell telephone companies.
traction of lines and regions from grey tone line drawing images,
Pattern Recognition, vol. 17, pp. 493-506, 1984.
[7] W. H. Chen, W . K. Pratt, E . R. Hamilton, R . H. Wallis, and P. J .
Capitant, Combined symbol matching facsimile data compression
system, Proc. ZEEE, vol. 68, pp. 786-796, 1980.
[8] F. M . Wahl, M. K. Y . Wong, and R . G . Casey, Block segmentation
and text extraction in mixed text/image documents, Cornput. Vision,
Graphics, Image Processing, vol. 20, pp. 375-390, 1982.
191 H. Bley, Segmentation and preprocessing of electrical schematics Rangachar Kasturi (M82) was born in Banga-
using picture graphs, Cornput. Vision, Graphics, Image Processing, lore, India, in 1949. He received the B.E. degree
vol. 28, pp. 271-288, 1984. in electrical engineering from Bangalore Univer-
[IO] L. A. Fletcher, Text string separation from mixed text/graphics im- sity in 1968 and the M.S.E.E. and Ph.D. degrees
ages, M.S. thesis, Dep. Elec. Eng., Pennsylvania State Univ., Aug. from Texas Tech University, Lubbock, in 1980
1986. and 1982, respectively.
111) A. Rosenfeld and A. C. Kak, Digital Pictureprocessing, vol. 2, 2nd Dr. Kasturi is an Associate Professor of Elec-
ed. New York: Academic, 1982. trical Engineering at the Pennsylvania State Uni-
1121 J. P. Foith, C. Eisenbarth, E. Enderle, H. Geisselmann, H. Rings- versity, where he was an Assistant Professor from
hauser, and G. Zimmermann, Real-time processing of binary im- 1982 to 1986. From 1978 to 1982 he was a Re-
ages for industrial applications, in Digital Image Processing Sys- search Assistant at Texas Tech University and was
tems, L. Bok and Z . Kulpa, Eds. Berlin: Springer-Verlag, 1981. engaged in research in multiplex holography and digital image processing.
[I31 W. K. Pratt, Digital Zmage Processing. New York: Wiley, 1978, From 1976 to 1978 he was the Engineering Officer at the Visvesvaraya
pp. 523-525. Industrial and Technological Museum, Bangalore, and from 1969 to 1976
he was with the Bharat Electronics Ltd., Bangalore, as a Research and
Development Engineer. His current research interests are in the applica-
tions of image analysis and artificial intelligence techniques for map data
Lloyd Alan Fletcher was born in Birmingham, processing, graphics recognition, and document analysis. He has directed
England, in 1962. He received the B . S . degree in several projects sponsored by NSF, Digital Equipment Corporation, AT&T,
physics with microelectronics and computing from and the Applied Research Laboratory. He has published a number of papers
the University of Leicester, England, in 1984. in journals and conference proceedings. He is the author of a book chapter
From August 1984 until August 1986 he attended and is coeditor of a book on image analysis applications to be published by
the Pennsylvania State University, where he was Marcel Dekker.
a teaching and research assistant, receiving the Dr. Kasturi is a member of OSA, SPIE, Eta Kappa Nu, and Sigma Xi.
M.S. degree in electrical engineering in 1986. At
Penn State his research interests included com-
puter vision and digital image analysis.
Since September 1986 he has been a Member