IEEE TRANSACTIONS ON REHABILITATION ENGINEERING 1

Automatic Visual to Tactile Translation, Part I:

Human Factors, Access Methods and
Image Manipulation
Thomas P. Way and Kenneth E. Barner
Applied Science and Engineering Laboratories
University of Delaware/Alfred I. duPont Institute
Newark, Delaware 19711
(302) 651-6830
E-mail: fway,[email protected].
Abstract | This is the rst part of a two-part paper that video and animation are not directly accessible to the blind
motivates and evaluates a method for the automatic conver- computer user.
sion of images from visual to tactile form. In this part, a
broad-ranging background is provided in the areas of human The sense of touch is relied upon frequently by blind per-
factors, including the human sensory system, tactual per- sons in lieu of sight. One common method of presenting vi-
ception and blindness, access technology for tactile graphics sual images in a touchable or tactile fashion is through use
production, and image processing techniques and their ap- of tactile graphics [49]. Tactile graphics provide a raised
propriateness to tactile image creation. In Part II, this back-
ground is applied in the development of the TACTile Image representation of such visually useful materials as maps,
Creation System (TACTICS), a prototype for an automatic graphs and other simple drawings. By current practice,
visual-to-tactile translator. The results of an experimental these are prepared by a sighted person individually and
evaluation are then presented and discussed, and possible
future work in this area is outlined. by hand. This preparation is neither timely nor ecient.
Keywords | blindness, image processing, microcapsule pa- Timeliness, while not a major issue for infrequently chang-
per, tactile graphics, tactile imaging ing items, such as maps, is a consideration for the large
volume of frequently changing visual information [20], such
as that available via the Internet and from other sources.
I. Introduction

A
Tactile imaging is the process of turning a visual item,
CCESS to visual information can widen the avenues such as a picture, into a touchable raised version of the im-
of professional and social interaction for blind per- age so that this tactile rendition faithfully represents the
sons. This is often accomplished through a manual process original information. Properly done, tactile imaging pro-
that translates a visual representation into a corresponding vides access for blind persons to visual information that is
tactile form. Involvement of a sighted person in this con- inaccessible via other means such as audio or textual de-
version step generally is necessary, however, limiting the scription. Tactual perception, the physiological capabilities
autonomy of the blind person. Further, this conversion is of the human sensory system to explore and discern via the
a time-consuming eort involving the use of glue, string, sense of touch, is well understood. Factors such as the size
scissors, cardboard and other craft materials, tracing pa- and shape of the ngertip, temporal and spatial response
per and marking pens, or computer-aided drawing pack- of the nerve receptors in the skin, and incorporation of
ages to produce a tangible representation of the original kinesthetic, or haptic, cues must be considered. These fac-
image. Although worthwhile, such an approach is neither tors limit the size and detail of tactile images to within the
timely nor easily reproducible, and clearly necessitates the response ranges of these various factors [49], [72].
involvement of a specially skilled sighted individual in the The way in which the mind perceives and classies im-
process. ages is a well-studied area, one in which a number of the-
Computers excel at displaying information via multiple ories have developed. Among these, perhaps the most ac-
media, including the CDROM and ubiquitous Internet. cepted view is that of human memory being arranged hier-
The omnipresence of the computer in everyday life pro- archically from general to specic in terms of one or more
vides ready availability to a myriad of graphical, textual qualities of the object being perceived. Whether the infor-
and auditory information for sighted and blind individu- mation is visual or tactile, the brain uses this same general
als alike. For blind computer users, text-based information framework for classication [14], [36], [40]. Thus, produc-
is output as synthesized speech or as braille via a special ing usable tactile images from photographs is a challenge
purpose printer or display. The surging prevalence of the requiring a careful balance of resolution, size, shape and
graphical user interface (GUI), however, introduces severe detail. Having too much detail in a tactile image will re-
impediments for the blind community. Pictures, drawings, sult in much of its content being lost, actually degrading its
2 IEEE TRANSACTIONS ON REHABILITATION ENGINEERING

clarity and utility due to an information overload of sorts. analyzed for signicance, and anecdotal evidence is added
This overload results from limitations of tactual percep- to support a discussion of their potential import. Finally,
tion, particularly the physiological disparity between the future directions in which this work may lead are outlined.
resolution of the human eye and ngertip. Including too
little detail will result in a tactile image that may not feel II. Human Factors
like anything more than a simple shape, not adequately The ecacy of a method for automatically converting vi-
representing the original image at all [37]. This ambiguity sual information into tactile information necessarily is de-
is due to the manner in which the brain categorizes what pendent upon a variety of factors, which are reviewed in
it perceives, in this case classifying tactually indistinguish- this section. To guide the design of such a system, an un-
able items as the same, even though the unprocessed visual derstanding of the human factors of sensation and percep-
originals may have been quite dierent. tion, including how the sense of touch compares to the sense
This two-part paper explores the possibility of automat- of sight, is important. There are lessons to be learned from
ically converting some forms of visual information into tac- past and current techniques for tactile graphic production
tile information. The means for testing this idea is the and other non-visual methods used by blind persons to ac-
development of a composite software/hardware system for cess computer-based information. The medium for the de-
automatic translation of electronic images into tactile form. scription of visual information that is under consideration
In this system, an aggregate process comprised of a se- in this paper is the computerized image. How such images
quence of image processing algorithms is applied to an im- are represented and the techniques that can be used to op-
age to produce a simplied version of the original. This car- erate upon them are explored, and their correspondence to
icaturized image is subsequently output in a raised tactile human tactual perception is considered. The background1
graphic form on microcapsule paper, suitable for display to provided in this section will be used to motivate the pro-
a blind person. totype system and experimental protocol detailed later.
In the rst part, topics in tactual perception, the human
sensory system in general, tactile graphics production and A. The Human Sensory System
image processing are examined. To provide access to visual The fundamental issue in presenting visual information
information for blind persons, an understanding of how we, in a meaningful tactile form is the understanding of some
as humans, interface with the world around us is vital. Per- basics of human sensory perception. By reviewing how the
ception at the tactual and mental levels and related human human sensory system collects and comprehends informa-
performance parameters are discussed, and the visual and tion and what the limits are to the type and amount of
tactual senses are contrasted. Relevant statistics regard- information the senses can process, it may be possible to
ing the blind population are presented, and an overview of identify factors that can play a role in the conversion of
blind computer user interface technology is provided. The information intended for one sense to a form suitable for
techniques for tactile graphic production are reviewed, as another sense.
is current research in this area. Cumulatively, this back- Humans receive all of their information about the world
ground propels a further discussion of image processing al- around them using one or more of ve senses [14]. The
gorithms that can assist in the perception of information Gustatory Sense provides information on taste qualities
that is converted from visual to tactual. This broad array such as sweet, salty, sour and bitter. Often working in
of background information is provided to justify a hereto- conjunction with taste is the Olfactory Sense, which pro-
fore unexplored combination of factors and theories from vides smell information. The Auditory Sense, our hearing,
these areas, all of which play formative and vital roles in allows us to receive auditory information such as music,
the motivation of this research. speech and noise. The Tactual Sense is comprised of touch
Presented in the second part is an evaluation of the TAC- and kinesthesis, providing information about such physical
Tile Image Creation System (TACTICS) [85], [86], which world qualities as temperature, perception of texture, po-
attempts the automatic conversion of images from visual sition and motion. Finally, the Visual Sense, our sense of
to tactile. This prototype system provides access to previ- sight, is how we receive visual information including color,
ously inaccessible visual information using image process- brightness, depth of eld, and motion.
ing and tactile graphics production techniques. The goal of The bandwidth of a sense refers to the capacity of that
this system is to free the blind computer user from reliance sense to receive and perceive information. Studies show
upon a sighted individual to prepare custom tactile graph- that vision, as one might intuitively expect, is our high-
ics, or tactics [25], and to overcome the considerable time est bandwidth sense, followed by hearing and touch (Ta-
delay in doing so. The specic techniques used in this sys- ble I2) [39]. The Visual Sense is two orders of magnitude
tem are introduced and support is provided linking their better at carrying information than the Auditory Sense,
use to applicable theories of perception. The ecacy of which is two orders of magnitude better than the Tactual
these techniques, which involve the application of a num- Sense.
ber of image processing algorithms in various combination 1 See Table III for a summary of various parameters related to tac-
and sequences, is evaluated in terms of discriminability, tual perception and that aected the design of TACTICS.
identiability and comprehensibility as measured in a se- 2 Table I states that the human ngertip processes vibrotactile sig-
ries of experiments. The results of these experiments are nals at a rate of 102 bits=sec. The results of previous research indicate
WAY AND BARNER: AUTOMATIC VISUAL TO TACTILE TRANSLATION 3

The Gustatory and Olfactory Senses are much more cernible displacement of a point on a surface. For highly
prone than the others to the eects of adaptation, and are smooth surfaces and under carefully controlled laboratory
not ecient at carrying information at a rate anywhere conditions, a 2-micron high point can be felt using active
near that of even the Tactual Sense. Adaptation refers to touch [44]. The height of a braille dot, an easily discernible
the tendency of a sense to grow accustomed to a stimulus, object, is in the range 0.02 - 0.05cm [21]. This is a gener-
thereby becoming less sensitive to it over time. Taste and ally acceptable range of heights for tactile graphics, with
smell are prone to adaptation and have comparatively slow heights at the upper end of the range naturally provid-
recovery times, while the other three senses have speed- ing relative improvements in perceptibility [49], much as
ier recovery times that are roughly proportional to their brighter lighting or higher volume can, to a point, improve
bandwidths. As the highest bandwidth and most resilient perceptability in the visual and auditory domains. The
sense, vision is arguably of the greatest importance among limiting factor for the height of tactile graphics is inherent
the senses, and therefore the hardest to do without. By in the media in which they are produced.
comparison, the other senses have lower to much lower in- Spatial tactile discrimination has been measured using
formation capacities which makes the problem of sensory square-wave gratings of varying groove amplitudes and sep-
substitution for vision a dicult one to address [14], [37]. arations under conditions of active exploration [33], [49],
The implications for development of a vision substitu- [72]. Sequences of gratings were presented to the distal
tion system are signicant by virtue of this large bandwidth pad of the right index nger in both the same and orthog-
disparity. Visual information cannot simply be mapped di- onal orientations to the axis of the nger. Observers noted
rectly to the auditory or tactual domains, but clearly must dierences in orientation of the grooves, which revealed
be reduced by some bandwidth correlated scaling factor. the distance at which orientation of grooves became indis-
Further, this scaling must preserve the meaning of the orig- criminable. This study demonstrated that the minimum
inal visual information to be useful. tactually discernible grating resolution is 1.0mm, and that
such discrimination improves linearly as the grating width
B. Tactual Perception increases above 1.0mm. This result is due to the forward
Tactual perception primarily refers to active exploratory masking eect of one stimulus upon perception of subse-
and manipulative touch. Study of the physiological fac- quent stimuli. The cutaneous receptors in the skin require
tors involved in tactual perception is important if one is to a period of time to recover after cessation of one stim-
gain an understanding of how best to create tactile images. ulus before correct sensing of a subsequent stimulus can
For a tactile image to be useful, a blind person must be begin [48].
able to explore it with the sense of touch, usually the n- Taken together, these factors appear to indicate that the
gers, and extract some content information. Thus, limits resolution of a tactile image should be somewhat ner than
to tactual perception, such as resolution of the human n- 1 dot=mm to produce a relatively smooth feel to the image,
gertip, image scale as a factor of comprehension, and how while resolutions much lower than this seem to provide lit-
the mind processes such information are important consid- tle or no benet to tactile perceptibility. For comparison,
erations [48], [49]. a resolution of 1 dot=mm equals 25.4 dots=inch, and the
The basic physiology of the human skin denes limits to resolution of a standard laser printer is at least as ne as
the ability of our sense of touch. Of particular importance 300 dots=inch.
to tactile graphics are the dierence limen and its relation C. Tactile Pattern Perception
to temporal response thresholds and masking phenomena.
The dierence limen is the minimum statically discernible The visual sense responds well to minute dierences in
displacement between two points such that the points are stimulus, while the sense of touch tends to need greater
distinct. In eect, this is tactile resolution, which for the variation in stimulus patterns to succeed in perceptual
skin of the ngertip is approximately 2.5mm. When stati- tasks [38], [49]. Although touch can discriminate and recog-
cally felt, two points closer than this distance tend to feel nize complex tactile patterns [37], such perception involves
like one point, whereas two points farther apart than this a number of complicated cognitive processes [41].
feel like two distinct points [72]. This gure indicates that There is strong basis for the supposition that spatial in-
the resolution of the ngertip is much lower than the hu- formation, which includes graphics, is stored in the visual
man eye. Therefore, we can safely say that tactile images cortex portion of the brain [40]. This mechanism is similar
require lower resolution than visual images. The denitive for sighted and blind persons, regardless of whether this in-
work on this two-point threshold, including its use as an formation is gathered using the sense of sight or touch. Re-
indicator of the relative spatial resolution as a function of search indicates that the ability to store and subsequently
body locus, is in [87]. retrieve tactually perceived spatial information can vary
Spatial sensing incorporates what we know about static greatly from individual to individual. This variation de-
sensing, embellished with further measurements of sen- pends to a signicant degree on the level of visual memory
sory abilities taken during motion of the nger [70]. Re- a blind person possesses, as often determined by the age of
lated to the two-point dierence limen is the minimum dis- the onset of blindness. There is comparatively little varia-
tion in such ability among the sighted population [67]. The
this rate to be no more than 101 bits=sec [15], [26], [42]. storage and retrieval of spatial information is believed to be
4 IEEE TRANSACTIONS ON REHABILITATION ENGINEERING

organized in a hierarchical fashion in the brain, which clas- such inherently three-dimensional information when it is
sies information based on gross characteristics rst, fol- mapped onto a two-dimensional display [49].
lowed by detailed characteristics [6], [80]. Although the res-
olution of the sense of touch degrades slowly with age [61], E. The Blind Population
which unfortunately equates with a statistical rise in blind- The American Foundation for the Blind recommends
ness [67], [69], experience with tactile graphics can make that the term blind be reserved for individuals with no us-
up for this slight loss of touch sensitivity [37], [84]. able sight whatsoever, while low vision, visually impaired
The method typically used by a blind person to explore or partially sighted can be used to describe those with some
a tactile graphic tends to support the hierarchical view of usable vision. These terms coincide with standard medical
human spatial memory. The exploration by a blind person diagnostic guidelines which divide visual impairment into
of a tactile graphic generally is performed in two stages. two classications: no light perception (NLP) and light per-
First, the entire image is explored as a whole, providing a ception (LP). An individual with corrected visual acuity of
general tactile overview. Second, the details of the tactile 20=200 in the better eye or a visual eld of 20 degrees or
image are explored. Research has veried this method- less in the better eye is considered legally blind. A blind
ology [29] and has shown that this technique is used by person is either congenitally blind , being blind from birth
blindfolded sighted persons as well. These results indicate or during the rst ve years of life and possibly lacking
that the concept of a hierarchical structure of the human visual memory, or adventitiously blind, with blindness be-
spatial memory is a reasonable assumption. ginning after the age ve and with the probable presence
It is important to note that the acuity of the touch sense of visual memory. Visual memory means the ability to
is comparable to blurred vision in similar tasks [1], [47]. classify and remember objects we perceive in terms of vi-
The signicance of this relationship is that any tactile rep- sual characteristics, such as shape, size, color, position and
resentation of visual information, based on what we already perspective [67].
know about tactual perception, should be suciently sim- There exist numerous misconceptions regarding blind
ple to make up for this reduced level of acuity [16], [20], persons [31], [53], [67]. Positive misconceptions are that
[49], [72]. This result supports our choice of pursuing meth- blind people are exceptionally musical, possess extraordi-
ods of image simplication in producing tactile images from nary senses of hearing and touch, and are highly intelligent.
their visual counterparts. Negative misconceptions include suppositions of helpless-
ness, dependence, laziness and lack of intelligence. Of par-
D. Aiding Comprehension ticular relevance is the supposed increased sense of touch.
Comprehension of a tactile display is increased when the Touch sensitivity varies little from person to person, with
reader is somehow clued in to what will be felt [20]. Just no statistical dierence between the sighted and blind pop-
as one expects photographs in a newspaper to have an as- ulation [50]. However, it does seem reasonable that a blind
sociated caption, so too would one reasonably expect that person may be more accustomed to relying on the sense of
the comprehensibility of a tactile image would be enhanced touch and interpreting tactual information [3], [37].
by including some associated textual information. This en- Statistics released by the World Health Organization in
hancement can be accomplished using standard techniques, 1987 estimate that there are 30- to 40-million blind peo-
such as by incorporating braille text with an image or by ple in the world [67]. According to 1989 statistics from
using speech output from a computer speech synthesizer to the National Society to Prevent Blindness, approximately
add information and increase comprehension. 500,000 U.S. residents are legally blind [67]. Of those g-
In a photograph, information about the relative depth ures, roughly ten percent are totally without sight [69].
within the eld of view of objects is provided by masking, The increase in the general population's reliance upon
shadows and size [14], [40]. This information is not readily the computer carries over to the blind population as
discernible in a tactile format and is a factor which can in- well [8]. As the number of computer users continues to
hibit the comprehensibility of a tactile image. One surpris- grow quite rapidly, any precise count of users would obvi-
ing side eect of congenital blindness on comprehension is ously be out of date even before it was written down. How-
the relative insensitivity to orientation of the tactile graphic ever, what is certain is that this number is suciently large
being touched. Where blindfolded sighted subjects in one to support an assertion that blind computer users make up
study were confused by a rotated or non-upright tactile a sizable group. It is worth noting that the availability
graphic representation of a known object, blind subjects and aordability of synthetic speech output via computer
suered little confusion. These blind subjects were quite has broadened access to information for this population as
facile at mentally rotating the spatial information perceived compared to braille access to the same information.
from the graphic representation, performing much better at According to the American Printing House for the Blind
comprehension tasks than the sighted subjects under the (APH), of the blind population residing in the United
same conditions [12]. States and of reading age, fewer than 16 percent are uent
Representing depth and perspective in a tactile image is in braille, while worldwide the gure is lower still [84]. An-
dicult, if not impossible, using a two-dimensional tactile other study cites the braille uency rate among blind and
display medium. Further, the congenitally blind individ- visually impaired computer users at 10 percent [27]. While
ual lacks a visual frame of reference for interpretation of these low braille literacy rates are discouraging, there is
WAY AND BARNER: AUTOMATIC VISUAL TO TACTILE TRANSLATION 5

some reason for optimismin the future. In a study of school convey successfully the content of the image. Clutter or
systems for blind children, more than one third of the stu- an overabundance of detail in a tactile image can detract
dents were found to be uent in braille, although audio from its usability and hamper one's ability to understand
output, either in the form of recorded books or speech syn- its content [38], [49]. Thus, it is important to simplify
thesis, was still the mode of choice at the time of the study complex images in the editing step of the process of con-
(Table II3 ) [84], [90]. verting them to tactile images. Experience shows that a
tactile graphic that is too large or too small detracts from
III. Access Technology for Blind Computer comprehensibility as well [89]. The size of a tactile image
Users should be kept within a hand span, or roughly 3in to 5in
Blind persons have a great many means for accessing on a side.
textual and visual information [7], [10], [11], [13], [19], [20], Transferral entails placing the image onto some tactile
[25], [24], [27], [43], [54], [78]. A number of these methods output medium. A picture is rst traced on tracing paper,
already do or can be adapted to provide blind computer and then is transferred to the tactile display material using
users with access to graphical information. Many tradi- carbon paper and retracing. Other methods for transferral
tional methods of access, such as braille output in one form include the pantograph, which is an instrument consisting of
or another, are, and continue to be, widely used. Their e- four arms jointed in parallelogram form. It is adjustable to
cacy is unquestioned. Some relatively recent developments, produce tracings of smaller, the same, or larger sizes. Using
such as speech output, are also eective and quickly merg- grids to scale images is also a common technique, as is
ing with traditional methods to create new standards for use of the enlargement capabilities of modern photocopier
access. Research is active in the development of dynamic machines.
and refreshable tactile displays [11], [19]. Innovations in the The production step is where the physical tactile graphic
materials and techniques used to display visual informa- is produced. There are numerous methods considered stan-
tion in a non-visual fashion are achieving some success [22], dard; without exception, all require the intervention of a
[82]. These new methods show promise, although technol- sighted person to translate a visual image into a tactile one.
ogy continues to lag behind concept. There are a number of commonly used methods for tactile
The task of accessing visual information is one of map- graphic production [19], [20], [78], including the following:
ping information from the visual domain to that of one of Raised-line drawing boards: Designed to be used by
the other senses. Knowing that this is essentially an in- blind persons for producing raised-line drawings, this com-
formation volume-reduction problem, given that the band- mon tool is also useful for fast production of tactile ver-
width of each of the other four senses is signicantly lower sions of visual originals. A stylus produces a raised line
than that of vision, it is helpful to look at some of the more when drawn over a plastic lm, giving an instant tactile
successful approaches to tackling this problem before de- representation.
veloping additional solutions. These methods fall into the Tactile-experience pictures: This method is often
general categories of Static Tactile Graphics, Auditory In- used for young children. Pictures are constructed of a vari-
terfaces, Dynamic Tactile Graphics and Haptic Interfaces. ety of materials, including wood, plastic, cloth, sandpaper,
In addition to these available means, there is active research fur, and metal, which are glued to a sti cardboard backing.
in this area that is worth reviewing as well. This method involves individually fashioning each piece out
A. Static Tactile Graphics of the desired material and assembling the resulting pieces
into the tactile picture.
Methods for production of static tactile graphics are var- Buildup displays: Similar in method to tactile-
ied and usually require the intervention of a sighted per- experience pictures, buildup displays rely on multiple layers
son in their preparation [20], [78]. The process of convert- of paper to build up a raised drawing. Additional materi-
ing computer graphics to tactile graphics can be a labor- als, such as wire, string and even staples, may be added to
intensive and time-consuming one. There are three im- enhance the drawing.
portant steps in this process: (1) editing, (2) transferral Embossed paper displays: This technique reproduces
and (3) production. Consider any original two-dimensional a drawing on heavy paper using a collection of embossing
graphic, such as a pencil sketch, ink drawing, graph, dia- tools. A reverse view of a sketch is rst transferred to the
gram, illustration or printed picture. back of a sheet of embossing paper. The tools are then
For a tactile graphic display to be comprehensible, it used to trace the sketch, embossing it as a series of raised
must not contain too much information. General design dots.
guidelines, developed through years of practical application Braille graphics: Graphics embossing can be produced
and renement of technique, suggest that a tactile graphic more simply and speedily using a standard braille printer
should contain the least amount of information possible to connected to a computer. Operating in graphics mode, the
3 An important distinction is made in the study regarding the de- printer maps pixels of the original image to braille dots to
nition of totally blind. Note that a small percentage (approximately produce the embossed version of the picture. The resolu-
2%) of students classied as blind possessed enough residual sight to tion of this method is low, and to be eective, the original
make use of Large Type, either alone or in combination with Braille image must be a simple line drawing. This method has
writing. For purposes of the study, students with either extremely low
acuity or a narrow eld of view were classied as totally blind [90]. two distinct advantages: many blind computer users have
6 IEEE TRANSACTIONS ON REHABILITATION ENGINEERING

access to a braille printer and no sighted intervention is as the Tactile Image Enhancer (Figure 3). For expanding
required for its use. Hence, with the proper processing multiple pages, each exposed sheet of capsule paper must
techniques applied to images, as will be described in the be fed individually into the Enhancer.
discussion of TACTICS, it may be possible to utilize such When exposed to a heat source of 120-125 degrees Celsius
a printer to produce adequate tactile representations of pic- (248-257 degrees Fahrenheit), portions of the paper that
tures. are printed in black expand. The microcapsules beneath
Vacuum-forming method: This method, also known the black lines of a diagram absorb more heat than the
as \thermoforming," excels at producing multiple copies of other microcapsules and expand in diameter, raising the
a tactile graphic in a very durable format. It requires a drawing from the background (Figure 4).
raised master made of stable or unpliable material. Next, An added benet is that one can draw directly on the
the master is placed on a perforated tray in the vacuum- microcapsule paper, which then can be raised immediately.
forming machine. A sheet of thin plastic is fastened over The time taken to raise one drawing already on a sheet of
the master such that it forms an airtight cover. A heating microcapsule paper is approximately ten seconds. Even ac-
unit is placed over the plastic as air is sucked out from counting for printing from a computer, photocopying onto
below the master, deforming the now pliant plastic over the microcapsule paper, and subsequent raising, the entire
the master. Once cooled, the plastic sheet is a durable process is still reasonably fast. Instant raised lines can be
replica of the original. This process can take as little as one produced on capsule paper using a new heat-pen device
minute, which is acceptable for producing multiple copies. developed by Repro-Tronics.
Microcapsule paper: Referred to variously as \capsule Other methods: Numerous other methods exist for
paper," \swell paper" or \pu paper," this is a quick and the production of tactile graphics, although none are
economical way to produce tactile graphics. It is paper that widely used. For purposes of completeness we mention
has been coated with microscopic capsules of polystyrene only their names here. These additional methods in-
(Figure 1), each being 100m in diameter. clude relief maps, cork maps and graphs, nongurative pic-
There are two types of microcapsule paper available on tures, sewing-machine diagrams, embossed aluminum-foil
the international market. Flexi-Paper is a polyethylene- displays, movable-parts displays, annel-board diagrams,
based paper manufactured by Repro-Tronics, in West- magnetic-board diagrams, electroforming processing, nylo-
wood, New Jersey [64]. It is tan in color and is quite print, silk screening, the solid-dot process, foam-ink print-
durable under conditions of folding and crumpling. The ing, storm relief printing, and screen drawings. Exhaustive
Matsumoto Kosan Company of Osaka, Japan, produces coverage of all of the above techniques are available in a
a paper-based version [51], white in color, that provides variety of sources, including [11], [19], [20], [78].
for blind persons a more familiar sti feel resembling that These static display methods typically produce long-
of heavy braille embossing paper while being less resis- lasting, eective displays of static visual information. For
tant to the eects of folding than Flexi-Paper. Both are dynamic information, such as material displayed on a com-
comparable in price ($1.00 U.S. per sheet). With an puter screen, other access methods are more appropriate.
unexpanded capsule diameter of 100m, the unexpanded
resolution of both brands is therefore 104 capsules=cm B. Auditory Interfaces
(2.54 104 capsules=in). The capsules expand upward This paper focuses on the production of tactile graphic
and outward consistently to a diameter (height) of 0.2mm output of information of a primarily graphical or visual
to 1.0mm, yielding an expanded resolution of 10 to 50 nature, but it is worth noting that auditory output is the
capsules=cm (25 to 127 capsules=in). In practical obser- method of choice for display of textual informationfor blind
vations in the laboratory, the typical expanded diameter is computer users [11], [27]. While there is a wide variety
0.3mm and typical expanded height is 1.0mm. of methods for production of tactile graphics, output of
To benet from this expanded resolution, a printer computer-generated speech is more generic. Screen review
should have a resolution of at least 127 dots=inch, the software is used by the blind computer user to explore the
best possible resolution of expanded capsule paper based on textual material and to select the desired passage. Typi-
manufacturers' specications. Thus, a typical laser printer cally, the software sends the text it encounters to a hard-
with a resolution of 300 dots=inch is entirely adequate for ware device, such as a speech-synthesis card added as an
initial output of the image to be expanded. The amplitude enhancement to a computer, for conversion from text to
of this expansion is aected by the temperature of the heat- speech [76]. One big benet of speech output is that users
ing element, with higher temperatures producing slightly who cannot read braille can use it; in addition, it is gen-
more pronounced expansion. erally quite aordable. Reliable speech synthesizers are
Original graphics are photocopied onto the capsule paper available for most computers, and the quality of speech is
using a standard oce copy machine (Figure 2). Graphics typically quite good. Perhaps the most attractive feature
can also be applied to the microcapsule paper using ink of the screen review and speech synthesis output method is
pens, markers and other drawing implements. The only adjustable speaking speed, enabling a blind person to listen
requirement is that the graphic be rendered in black. Once at 300 words=minute or more [11], [71], [78], a speed that
the image is applied to the microcapsule paper, it is in- is quite competitive with typical sighted-reading speeds of
serted image side up into a heating machine, referred to 250 to 500 words=minute [18].
WAY AND BARNER: AUTOMATIC VISUAL TO TACTILE TRANSLATION 7

The Nomad is an example of a multimodal device, com- vibrating pins. The idea of the system was eventually to
bining static tactile graphics with audio output. A tactile produce a system by which a blind person could wear a
graphic, such as a map, is produced and axed to the video camera and backpack display and actually maneu-
display surface of the Nomad. This surface is addressable ver through the world using the vibrating representation
via computer; and each region can be mapped to sounds of what the camera saw for guidance. The technique may
that will play in response to the associated region being have been ahead of its time, being bulky and noisy, even
touched. The Nomad is well suited to museum displays and by early 1970s standards. Modern technology may yet pro-
shopping-mall maps but requires assistance from a sighted duce such a system for independent, walk-around vision
person for conguration [19]. replacement [7], [13], [17].
Research is underway at the University of Wisconsin ex-
ploring the use of an audiotactile snapshot approach [83]. D. Dynamic Tactile Display Research
This technique combines computer technology, a touch- Enabling blind persons to access visual data on a com-
screen interface, tactile representations of the computer puter meaningfully is an area of vigorous research. Some
screen and audio output, to provide multimodal access for of the more pertinent projects from the present and near
blind computer users. It is still in the experimental stage, past include:
but some successful tests have been performed with a Win-  A virtual tactile tablet incorporating a vibrotactile dis-
dows implementation on a personal computer. play module demonstrated that increasing a graphic's
size and its display resolution improved recognition, while
C. Dynamic Tactile Interfaces merely varying the geometric complexity of a graphic did
Currently, the only dynamic tactile display device in not dramatically eect object recognition [89].
wide use is the Optacon (Figure 5a). It is a vibrotac-  Experiments with a single-pin tactile mouse revealed that
tile display, comprised of a ngertip-sized matrix of 144 vi- immediate tactile feedback improved response times in GUI
brating pins, arranged in a 24-row, 6-column format. This navigation tasks [75].
display is contained in a portable case (8in  6in  2in,  The use of nickel-titanium shape-memory allow (SMA)
4.0 lbs) and is powered by one 5-volt, rechargeable, nickel- to provide actuation of a tactile display shows promise as
cadmium battery. Vibration is caused by piezoelectric lm the basis for a lightweight and portable display, although
bimorphs, which vibrate with varying amplitude at 230Hz the power consumed and the heat produced by such a dis-
in response to varying levels of current. Its use involves play are still high. Further, current shape-memory alloy
placing the nger of one hand onto the vibrotactile display suers from brittleness, slow response and recovery times,
pad and using the other hand to pass a scanning device and lack of long-term durability [28].
over the desired text or image.  A 64-solenoid, four-level, pin-based ngertip display,
The Optacon was designed as an alternative to braille used to investigate tactual comprehension improvement
for reading printed text; but reading speeds are slower (50 through representation of levels of graphics image inten-
words=minute after months of training and practice) than sity by varying pin heights on the display [23].
with braille (104 words=minute), and much slower than  A virtual tactile computer display which uses electrome-
with synthesized speech output (300+ words=minute) [19], chanically actuated pins in a rectangular tactile array com-
[21], [81]. The price of a new Optacon, in the neighborhood parable in size to the sensing area of the ngertip [34].
of $4,000.00 U.S., is also an issue for some [19], [73]. As of  The use of polymer gels, or electrorheological uids, for
the writing of this paper, the company which produced the fabrication of actuators which then conceivably could be
Optacon, Telesensory, has discontinued production; and used in the development of a tactile display. Such uids be-
negotiations are underway with another company, Blaise come rm when current is passed through them and could
Engineering, to continue production in the future [74]. also serve as the basis for a direct-touch, deformable tactile
During use, the pins of the Optacon display react inde- display [22], [56], [57], [60].
pendently in a one-to-one mapping of pixels, or groups of  Past research delved into electrocutaneous stimulators,
pixels, to pins in response to an image or text passed under which delivered tiny electrical shocks to the skin, and air
the lens of the scanner. Black regions of the scanned item jet stimulators, which replaced the pin array with an ar-
cause pins to vibrate while white regions inhibit vibration. rangement of tiny holes where pus of air are aimed at
Thus, a letter, line or picture feels like a vibrating replica the skin [17]. Neither of these methods was particularly
of the original [73] (Figure 5b). However, the vibrating dis- successful; these two methods are generally accepted by
play produces a noticeable amount of buzzing noise, and the mainstream research community as unworthy of fur-
the vibration itself tends to temporarily dull the sense of ther consideration.
touch on the nger resting on the display after a period of
use. E. Haptic Interfaces
In addition to the Optacon was the Tactile Vision Sub- The term haptic refers to the proprioceptive, or posi-
stitution System (TVSS ), which used a similar tech- tional, sense, which is an extension of touch [35]. Thus, a
nique to display a vibrating representation of an image on haptic interface can represent three or more dimensions
a user's back [4], [88]. The image was captured by a tele- whereas a tactile display provides only two dimensions.
vision camera and sent to a more widely spaced array of Haptic interfaces are an important display method in vir-
8 IEEE TRANSACTIONS ON REHABILITATION ENGINEERING

tual reality systems, capable of reproducing a sense of po- The basic unit of the computerized image is the pic-
sition in space, interaction of forces, and even textures. Of ture element or pixel [46]. For images represented solely
course, the original information must be multidimensional as shades of gray, each pixel is assigned a single value, typ-
as well, often generated by math-graphing packages or cus- ically an 8-bit integer. Thus, such an 8-bit grayscale image
tom graphing software. has an intensity range of 256 levels of gray, with 0 typically
Examples of this highly active area of research include indicating black and 255 indicating white. Similarly, color
development of a method for display of graphs of mathe- images typically have three such 8-bit intensity levels as-
matical functions and scientic data using a three-degree sociated with each pixel, one each for the R ed, G reen and
of freedom device called the PHANToM [25], [24], [52], pro- B lue components4 .
tein molecule docking simulations [10], three dimensional For purposes of this research, we consider primarily com-
volume haptization [30], and successful experiments in sim- plex computer images, quantized representations of pho-
ulating textures with an enhanced joystick device [54], [55]. tographs, electron micrographs, individual video images,
These devices are generally very expensive ($10,000.00 etc., as these present the greatest diculties when cre-
U.S. and up) and so are still relegated to a small number ating a tactile representation. Simple images, such as
of research facilities. It is hoped that eventually aordable sketches, diagrams, and line drawings often can be con-
haptic interfaces will be readily available, providing blind verted straightforwardly into tactile form. Complex images
computer users with an even greater ability to explore tra- are typically comprised of a broad and unpredictable mix-
ditionally visual information physically. An in-depth study ture of shape, color, intensity, and other real-world com-
of haptic interfaces is beyond the scope of this work, al- plexities, presenting the most signicant challenges to ac-
though progress in this area is clearly important to note. cess by the blind computer user.
An extensive bibliography on this topic is available in [55]. Image processing is a broad term describing the algo-
rithmic transformation of an image from one form to an-
F. Moving Toward Eective Tactile Display of Graphics other [63]. Processes are divided into general categories of:
(1) point processes, (2) area processes, (3) frame processes
Audio output is not a solution for most graphics prob- and (4) geometric processes [46].
lems because of the diculty of extracting meaning from 1. Point processes are the simplest and most frequently
a picture, known as the Image Understanding Problem [6], used of the image processing operations. A point process is
[9], [65], [68]. In order for synthesized speech output to pro- an algorithm that modies a pixel's value in an image based
vide adequate and automatic access to an image, the image solely upon that single pixel's value or location. Common
would rst have to be understood by the computer, an un- point processes are image brightening, negative images,
likely occurrence at present. The most promising direction image thresholding, image contrast stretching and image
for research is toward creation of a refreshable tactile dis- pseudocoloring.
play. Such a display would be the tactile equivalent of a 2. Area processes use groups of pixels surrounding a cen-
standard computer screen, or cathode ray tube, providing tral pixel of interest to derive information about an image.
direct access to the graphical contents of the computer. This group of pixels, often referred to as a neighborhood, is
For such a dynamic display to be usable by blind persons, examined in some algorithmic fashion as a group. This ex-
attention must be paid to how graphic material is to be amination, for instance, can determine the brightness trend
displayed. Clearly, the ngertip possesses a much lower information or spatial frequency, with the result utilized in
resolution than the eye, so complex visual information must determining a new value applied to the central pixel of the
be simplied somehow. Developing a system for performing neighborhood. Examples of area processes include edge
such simplication, including factors related to method, enhancement and detection, image sharpening, smoothing
eectiveness, usability, and future applicability, is the scope and blurring, and removal of random noise.
and direction of our research. 3. Frame processes use information from two or more
images, or video frames, together with a combination func-
IV. Manipulation of Images tion to produce a new image. Among the many practi-
An image is an alternative representation of some visual cal applications of frame processes are motion detection,
scene [46], [68]. These representations include sketches, background removal, image-quality enhancement and im-
drawings, photographs, computerized graphics and pic- age combination.
tures, and motion picture lm and videotape. We restrict 4. Geometric processes change the spatial positioning
our discussion to computerized images. or arrangement of pixels within an image based upon some
In order to create a computer image from some other geometric transformation. Typical operations performed
type, some form of quantization is performed. In this pro- by geometric processes include image scaling, sizing, rota-
cess, samples of the image are taken using a scanner or tion, translation and mirror imaging. Example uses include
digital camera at some regular interval and size, based on 4 Although color images are frequently represented in this RGB for-
the desired resolution of the nal quantized image. Each mat, numerous other representational schemes exist. Among the
sample is assigned a discrete value, or set of values, that most frequently used of these methods are: Cyan, Magenta and
represent the intensity or color of the sample as closely as Yellow (CMY), Hue, Saturation and Value (HSV), Hue, Saturation
and Lightness (HLS), Hue, Saturation and Intensity (HSI), and Hue,
possible [63], [68]. Chrominance and Intensity (HCI).
WAY AND BARNER: AUTOMATIC VISUAL TO TACTILE TRANSLATION 9

spatial aberration correction, image composition and spe- where there is a rapid change in intensity. The faster such
cial eects. a transition is made from light to dark, or vice versa, the
more likely an edge detection algorithm is to consider the
A. Image Processing Algorithms center of such a transition as an edge. Each pixel that is
There are a great many algorithms that process images found to be part of an edge is set to the color white, while
to produce a wide variety of eects [5], [6], [46], [62], [63], non-edge pixels can be left alone or assigned the color black
[66], [77]. We are concerned here with the eect more than using some thresholding function. A common version of
with the specic means. For a thorough understanding of this algorithm is the Sobel edge detector, which accom-
how the classes of algorithms we have chosen operate on plishes edge detection by using the scaled average of one of
images, and how they relate to our goal of image simpli- a 3  3 pixel neighborhood's horizontal or vertical direc-
cation, we present a brief and somewhat simplied intro- tional derivative, as rst described in [62]. The Sobel edge
duction to each of them. For purposes of this discussion, detection function makes use of two matrices, or masks,
we assume that an image is grayscale, although these algo- one each for the vertical and horizontal directions:
rithms have forms that work equally well for color images.
Since we are concerned neither with moving images nor ge- 2 3 2 3
ometric transformations, we do not consider frame or geo- ?1 0 1 1 2 1
metric processes; rather, we restrict coverage to a number V = 4 ?2 0 2 5 H = 4 0 0 0 5 (4)
of point and area processes. Detailed theoretical treatment ?1 0 1 ?1 ?2 ?1
of image processing techniques is available in [63], while an These masks are convolved over an image. In the case of
implementation-oriented approach is given in [46]. Sobel edge detection function FS , the two masks V and H
For clarity, the notation we use to describe images and are applied as follows for each point (m; n) in image X:
image processing algorithms is dened here. A grayscale
image X of overall width w and height h can be represented
by a two-dimensional array of points, each of which has a Am;n = Nm;n  V (5)
certain value, denoted by Xm;n , representing the brightness Bm;n = Nm;n  H (6)
or intensity of that point. X
A color image has a set of three intensity values, one A0m;n = u (7)
each for the red, green and blue components of each pixel, u2Am;n
associated with each position in the array. Formally, an
8-bit grayscale image is described by: 0 = X v
Bm;n (8)
v2Bm;n
q
X = 1  m  w; 1  n  h; Xm;n 2 f0; 1; : : :; 255g (1) FS (Xm;n ) = A0m;n 2 + Bm;n 0 2 (9)
The set of points N in a square region of width w0 sur- This is a very computationally expensive operation to
rounding a given point is the neighborhood of that point. perform, particularly for larger images, due to the neces-
For points that are closer than w 2?1 points to an image
0

sary 20 multiplications, 19 additions and 1 square-root op-

boundary, the neighborhood will include only those points eration per pixel. There are numerous methods described
falling within the image. The neighborhood of a point in the literature that can speed up this process.
Xm;n is denoted by the set: Blurring: Often referred to in the literature as low pass
ltering, blurring reduces the detail in an image by remov-
Nm;n = f w0 is odd; ing the high frequency component [68]. It accomplishes
max(m ? w 2?1 ; 1)  i  min(m + w 2?1 ; w);
0 0
this by using the values of all pixels in a neighborhood,
max(n ? w 2?1 ; 1)  j  min(n + w 2?1 ; h) :
0 0
assigning some function of those values to the center pixel.
Xi;j Application of either a Gaussian or averaging function are
g two common techniques to accomplish blurring. Averag-
(2) ing is the most straightforward and fastest technique and,
An algorithm a is represented by a mathematical func- considering the low resolution of the human ngertip, is
tion Fa that transforms an image X into a processed image sucient. The blurring function FB is described as:
Y , as follows: P
v
FB (Xm;n ) = jvN2Nm;nj (10)
Y = Fa (X) (3) m;n
The results of our implementations of the followingimage Applying this function to all pixels in an image produces
processing algorithms can be compared with the original a blurry version of the original image (Figure 6c). This
image in Figure 6a. is also described as the convolution over X by a blurring
Edge Detection: An edge detection algorithm at- mask or kernel. For example, the blurring algorithm used
tempts to locate and highlight edges in an image (Fig- in this research is accomplished with the following 3  3
ure 6b). These edges are simply the portions of an image kernel B:
10 IEEE TRANSACTIONS ON REHABILITATION ENGINEERING

2 3 As implemented, the adaptive version of the algorithm

1 1 1 performs the same steps as the K-means segmentation al-
B=4 1 1 1 5 (11) gorithm, with the dierence being that it operates to con-
1 1 1 vergence on each pixel in X before moving to the next pixel.
Segmentation: Images are generally comprised of one Thus, the K-means algorithm is performed on some subset
or more regions, dened as sections or segments of an im- or window of, and in complete isolation from, the image as
age whose members are all closely related by color or inten- a whole. Inspiration for this implementation is drawn from
sity. Segments are essentially the areas between edges. A portions of an adaptive segmentation algorithm that uses
common technique for locating segments is called K -means a Gibbs random eld model and a hierarchical approach
segmentation [45], [66], [79] (Figure 6d). In this algorithm, described in [59].
each pixel is assigned to one of some number K of dierent Negation: The negation of an image is produced by
groups, or intensity levels, based on its own intensity level. inverting the intensity value of each pixel in the image
This assignment serves to divide pixels with closely related (Figure 6f), assigning this new value to each pixel in turn.
intensities into like groups or clusters, producing an im- Negation is described by this simple function:
age that is segmented by intensity. A similar segmentation
can be performed based on color. Algorithmically, the K- FN (Xm;n ) = 255 ? Xm;n (14)
means segmentation applied to image X can be described Negation often is applied in conjunction with another
as follows [79]: algorithm. In the case of a strictly black and white or bi-
 Step 1. Choose K initial cluster centers z1(1), z2 (1), : : :, nary image with more black than white, subtracting the
zK (1). These can be chosen arbitrarily as, say, the intensity intensity of each pixel from the maximum reverses the eld
values of the rst K pixels in X, or evenly spaced across and makes foreground features such as edges black. This
the range 0 ? 255 as is implemented in our system. negation hopefully improves the legibility of a tactile im-
 Step 2. At the kth iterative step distribute the intensity age, specically when it is output on microcapsule paper,
values fXm;n g among the K cluster domains, using the since the black portion of the image raises while the white
relation: portion remains at.
Median Filtering: Median ltering is a method for re-
Xm;n 2 Sj (k); if jXm;n ? zj (k)j < jXm;n ? zi (k)j (12) moval of noise from an image [63]. Generally, noise in an
8i = 1; 2; : : :; K; i 6= j, where Sj (k) denotes the set of in- image is described as an individual pixel of greatly dier-
tensity values whose cluster center is zj (k). ing intensity, or outlier, compared to the typical pixel in a
 Step 3. From the results of Step 2, compute the new neighborhood. Dierentiating noise from minute detail, or
cluster centers zj (k +1); j = 1; 2; : : :; K, such that the sum ltering out noise while leaving the desired image intact,
of the squared distances from all points in Sj (k) to the is often not so straightforward [2], particularly when an
new cluster center is minimized. This is simply the mean image is complex. Performing edge detection on an image
of Sj (k), given by: tends to accentuate these outliers, whether noise or detail.
The median ltering algorithm sorts the intensity values
P of pixels in a neighborhood, assigning the median value of
X m;n the neighborhood to the center pixel. This is repeated for
zj (k + 1) = Xm;njS2S(k)
j (k)
; j = 1; 2; : : :; K (13) all pixels in the image, with the eect being a reduction in
j j
the number of outliers while preserving edges and non-noisy
It is from this manner in which each of the K cluster cen- portions of the image (Figure 7). An especially fast version
ters are iteratively updated with the average value for each of the median ltering algorithm can be found in [32]. The
cluster that the name \K-means" is derived. function FM for median ltering can be described as:
 Step 4. If zj (k + 1) = zj (k) for j = 1; 2; : : :; K, the algo-
rithm has converged and can be terminated. Otherwise, go FM (Xm;n ) = Median(Nm;n ) (15)
back to Step 2 and continue.
The fundamental drawback of this general statistical B. Applicability of Image Processing
analysis of, or histogram-based approach to, image segmen- Production of tactually perceivable tactile images bears
tation is the inherent disregard for spatial coherence [59]. some similarity to the challenges of the eld of computer
Adaptive segmentation attempts to take into account a vision. The aim of computer vision is automatically to
smaller portion of an image, producing a segmentation provide analysis of an image on which some decision can
based only on that portion. The eect of this process can be be based [9], [58]. Image processing techniques are invari-
to retain more of the original image information, producing ably used in this task to transform an image in such a way
a segmentation which more closely resembles the original as to produce some form of useful output. Similarly, the
(Figure 6e). This result often is achieved at some compu- aim of TACTICS is to present a visual image in a tactile
tational expense and many times produces a result only format such that it is useful in some way to an observer.
marginally better than a straightforward segmentation al- Image processing techniques would appear to be a natural
gorithm for purposes of image simplication and automatic approach. The limits to tactile resolution, and the under-
tactile graphics generation. stood importance of reducing to an essential minimum the
WAY AND BARNER: AUTOMATIC VISUAL TO TACTILE TRANSLATION 11

information presented to the ngertip, clearly calls for a References

simplifying transformation of complex images. [1] P. Apkarian-Stielau and J.M. Loomis. A comparison of tac-
In Part II, we demonstrate the use of these algorithms tile and blurred visual form perception. Perception and Psy-
chophysics, 18(5), 1975.
within the visual to tactile translation task. Algorithms are [2] G.R. Arce, N.C. Gallagher, and T.A. Nodes. Median lters:
applied in a sequence to an original computerized image, Theory and applications. In T.S. Huang, editor, Advances in
producing a simplied version. This simpler representation Computer Vision and Image Processing, volume 2. JAI Press,
1986.
is output and raised using microcapsule paper and the en- [3] N.C. Barraga. Sensory perceptual development. In G.T. Scholl,
hancing device, generating a tactile version of the original. editor, Foundations of Education for the Blind and Visually
We develop and evaluate this process in Part II. Handicapped Children and Youth: Theory and Practice. Ameri-
can Foundation for the Blind, 1986.
V. Summary [4] K.L. Beauchamp, D.W. Matheson, and L.A. Scadden. Eect of
stimulus-changemethod on tactile-imagerecognition. Perceptual
In Part I of this two-part paper we reviewed a variety of and Motor Skills, 33, 1971.
[5] P.J. Benson, D.I. Perrett, and D.N. Davis. Towards a quan-
issues in the areas of human factors, access technology for titative understanding of facial caricatures. In V. Bruce and
tactile graphics production, and image processing. A num- M. Burton, editors, Processing Images of Faces. Ablex Publish-
ber of the most pertinent of these parameters are collected ing Corporation, Norwood, New Jersey, 1992.
[6] I. Biederman. Human image understanding: Recent research and
in Table III. The background information presented in this a theory. Computer Vision, Graphics and Image Processing, 32,
part will motivate the development and testing of a pro- 1985.
[7] J.C. Bliss, M. Katcher, C.H. Rogers, and R.P. Shepard. Optical-
totype visual-to-tactile translation system in Part II. The to-tactile image conversion for the blind. IEEE Transactions on
design of our TACTile Image Creation System (TACTICS) Man-Machine Systems, MMS-11, 1970.
draws from this background, applicable image processing [8] L.H. Boyd, W.L. Boyd, and G.C. Vanderheiden. The graphical
techniques, and principles of psychophysics to attempt the user interface: Crisis, danger and opportunity. Journal of Visual
Impairment and Blindness, 84, 1990.
unsupervised conversion of pictorial information from vi- [9] R.D. Boyle and R.C. Thomas. Computer Vision: A First
sual to tactile form. In Part II, we evaluate the eectiveness Course. Blackwell Scientic Publications, London, 1988.
of TACTICS at producing tactile output that is discrim- [10] F.P. Brooks, M. Ouh-Young, J.J. Batter, and P.J. Kilpatrick.
Project grope - haptic displays for scientic visualization. In
inable, identiable and comprehensible. This evaluation 17th Annual ACM Conference on Computer Graphics and In-
is accomplished through a series of experiments, and an teractive Techniques - SIGGRAPH '90, volume 24 of Computer
Graphics, New York, August 1990. ACM.
analysis and discussion of the results of these experiments [11] D. Burger. Improved access to computers for the visually hand-
together with a number of anecdotal observations. Finally, icapped: New prospects and principles. IEEE Transactions on
potential directions for future research in this are proposed. Rehabilitation Engineering, 2(3), 1994.
[12] P.A. Carpenter and P. Eisenberg. Mental rotation and the frame
of reference in blind and sighted individuals. Perception and
Acknowledgments Psychophysics, 23(2), 1978.
Work on this project was performed at the University [13] C.C. Collins. Tactile television - mechanical and electrical im-
age projection. IEEE Transactions on Man-Machine Systems,
of Delaware Center for Applied Science and Engineering, MMS-11, 1970.
located at the Alfred I. duPont Institute, in Wilmington, [14] S. Coren and L.M. Ward. Sensation and Perception (3rd Edi-
Delaware. As part of the \Science, Engineering and Mathe- tion). Harcourt Brace Jovanovich, San Diego, 1989.
[15] J.C. Craig. Vibrotactile pattern perception: Extraordinary ob-
matics" project, funding was provided by the National Sci- servers. Science, 196, 1977.
ence Foundation, grant number HRD-9450019. Additional [16] J.C. Craig. Some factors aecting tactile pattern perception.
International Journal of Neuroscience, 19, 1983.
support was provided by the Nemours Research Programs. [17] J.C. Craig and C.E. Sherrick. Dynamic tactile displays. In
W. Schi and E. Foulke, editors, Tactual Perception: A Source-
book. Cambridge University Press, 1982.
[18] D. Crystal. The Cambridge Encyclopedia of Language. Cam-
bridge University Press, Cambridge, 1987.
[19] F. Deconinck and P. Verschueren. TIDE project 103 GUIB: A
model of the understanding of graphical information by blind
people. Technical report, Final Report, June 1993.
[20] P.K. Edman. Tactile Graphics. American Foundation for the
Blind, New York, 1992.
[21] E. Foulke. Reading braille. In W. Schi and E. Foulke, editors,
Tactual Perception: A Sourcebook. Cambridge University Press,
1982.
[22] J. Fricke and H. Baehring. Design of a tactile graphic I/O
tablet and its integration into a personal computer system
for blind users. Electronic Proceedings of the 1994 EASI
High Resolution Tactile Graphics Conference, Available from
http://www.rit.edu/easi/, 1994.
[23] S.F. Frisken-Gibson, P. Back-Y-Rita, W.J. Thompkins, and J.G.
Webster. A 64-solenoid, four-level ngertip search display for the
blind. IEEE Transactions on Biomedical Engineering, BME-
34(12), 1987.
[24] J.P. Fritz and K.E. Barner. Design of a haptic graphic system.
Proceedings of the RESNA '96 Annual Conference, 1996.
[25] J.P. Fritz, T.P. Way, and K.E. Barner. Haptic representation
of scientic data for visually impaired or blind persons. In Pro-
ceedings of the CSUN Conference on Technology and Disability,
1996.
12 IEEE TRANSACTIONS ON REHABILITATION ENGINEERING

[26] L.H. Goldish and H.E. Taylor. The optacon: A valuable device [53] B.S. Miller and W.H. Miller. Extinguishing `blindisms': A
for blind persons. New Outlook for the Blind, 68(2), 1974. paradigm for intervention. Education of the Visually Handi-
[27] D. Grith. Computer access for persons who are blind or vi- capped, Spring 1976.
sually impaired: Human factors issues. Human Factors, 32(4), [54] M. Minsky, M. Ouh-Young, O. Steele, F. Brooks, and M. Behen-
1990. sky. Feeling and seeing: Issues in force display. In Proceedings
[28] C.J. Hasser and J.M. Weisenberger. Preliminary evaluation of of the Symposium on 3D Real-Time, 1990.
a shape-memory alloy tactile feedback display. Advances in [55] M.D.R. Minsky. Computational Haptics: The Sandpaper System
Robotics, Mechatronics and Haptic Interfaces, 49, 1993. for Synthesizing Texture for a Force-Feedback Display. PhD
[29] R. Hinton. First introduction to tactiles. The British Journal thesis, Massachusetts Institute of Technology, June 1995.
of Visual Impairment, 9(3), 1991. [56] A.H. Mitwalli, S.B. Leeb, T. Tanaka, and U. Sinha. Polymer gel
[30] K. Hirota and M. Hirose. Simulation and presentation of curved actuators - status report. In Proceedings of the 29th Universities
surface in virtual reality environment through surface display. Power Engineering Symposium, Galway, Ireland, 1994.
In Proceedings - Virtual Reality Annual International Sympo- [57] G.J. Monkman. An electrorheological tactile display. Presence,
sium '95, Los Alamitos, California, 1995. IEEE Computer Soci- 1(2), 1992.
ety Press. [58] V.S. Nalwa. A Guided Tour of Computer Vision. Addison-
[31] L.T. Hoshmand. Blindisms: Some observations and proposi- Wesley Publishing Company, Reading, Massachusetts, 1993.
tions. Education of the Visually Handicapped, May 1975. [59] T.N. Pappas. An adaptive clustering algorithm for image seg-
[32] T.S. Huang, G.J. Yang, and G.Y. Tang. A fast two dimensional mentation. IEEE Transactions on Signal Processing, 40(4),
median ltering algorithm. Proceedings of the IEEE Conference 1992.
on Pattern Recognition and Image Processing, 1978. [60] R. Peier. Possible applications of polymer and photopolymer
[33] K.O. Johnson and J.R. Phillips. Tactile spatial resolution: Two- technologies to high resolution tactile graphics. Electronic Pro-
point discrimination, gap detection, grating resolution, and let- ceedings of the 1994 EASI High Resolution Tactile Graphics
ter recognition. Journal of Neurophysiology, 46, 1981. Conference, Available from http://www.rit.edu/easi/, 1994.
[34] P. Jubinski. VIRTAC, a virtual tactile computer display. Pro- [61] L. Petrosino and D. Fucci. Temporal resolution of the aging
ceedings of the Johns Hopkins National Search for Computing tactile sensory system. Perceptual and Motor Skills, 68, 1989.
Applications to Assist Persons with Disabilities, 1992.
[35] J.M. Kennedy. Haptic pictures. In W. Schi and E. Foulke, ed- [62] K.K. Pingle. Visual perception by a computer. In A. Grasselli,
itors, Tactual Perception: A Sourcebook. Cambridge University editor, Automatic Interpretation and Classication of Images.
Press, 1982. Academic Press, 1969.
[36] R.L. Klatzky. Human Memory: Structures and Processes. W.H. [63] W.K. Pratt. Digital Image Processing. John Wiley and Sons,
Freeman and Company, New York, 2nd edition, 1980. New York, 1991.
[37] R.L. Klatzky, S.J. Lederman, and V.A. Metzger. Identifying [64] Repro-Tronics Inc., Westwood, New Jersey. Setup and Operating
objects by touch: An \expert system". Perception and Psy- Instructions for the Tactile Image Enhancer, 1994. Product
chophysics, 37(4), 1985. specications.
[38] R.L. Klatzky, S.J. Lederman, and C. Reed. There's more to [65] E. Rich and K. Knight. Articial Intelligence. McGraw-Hill,
touch than meets the eye: The salience of object attributes for Inc., New York, 2nd edition, 1992.
haptics with and without vision. Journal of Experimental Psy- [66] A. Rosenfeld and L.S. Davis. Image segmentation and image
chology: General, 116, 1987. models. Proceedings of the IEEE, 67(5), 1979.
[39] K.J. Kokjer. The information capacity of the human ngertip. [67] J. Sardegna and T.O. Paul. The Encyclopedia of Blindness and
IEEE Transactions on Systems, Man, and Cybernetics, SMC- Vision Impairment. Facts On File, New York, 1991.
17(1), 1987. [68] R.J. Schalko. Digital Image Processing and Computer Vision:
[40] S.M. Kosslyn. Image and Brain: The Resolution of the Imagery An Introduction to Theory and Implementations. John Wiley
Debate. MIT Press, Cambridge, Massachusetts, 1994. and Sons, New York, 1989.
[41] L.E. Krueger. The psychophysics of touch. In W. Schi and [69] G.T. Scholl. What does it mean to be blind. In G.T. Scholl, edi-
E. Foulke, editors, Tactual Perception: A Sourcebook. Cam- tor, Foundations of Education for the Blind and Visually Hand-
bridge University Press, 1982. icapped Children and Youth: Theory and Practice. American
[42] Z. Kuc. A bidirectionalvibrotactilecommunicationsystem: Tac- Foundation for the Blind, 1986.
tual display design and attainable data rates. VLSI and Com- [70] A.S. Schwartz, A.J. Perey, and A. Azulay. Further analysis of
puter Peripherals, 1989. COMPEURO '89 - 3rd Annual Euro- active and passive touch in pattern discrimination. Bulletin of
pean Computer Conference. the Psychonomic Society, 6(1), 1975.
[43] M. Kurze, L. Reichert, and T. Strothotte. Access to business [71] R.S. Schwertfeger. Making the GUI talk. Byte Magazine, De-
graphics for blind people. Proceedings of the RESNA 17th An- cember 1991.
nual Conference, 1994. [72] C.E. Sherrick and J.C. Craig. The psychophysics of touch. In
[44] R.H. LaMotte and J. Whitehouse. Tactile detection of a dot on W. Schi and E. Foulke, editors, Tactual Perception: A Source-
a smooth surface: Peripheral neural events. Journal of Neuro- book. Cambridge University Press, 1982.
physiology, 56(4), 1986. [73] Telesensory Systems, Inc., Palo Alto, California. OPTACON
[45] A. Lev, S.W. Zucker, and A. Rosenfeld. Iterative enhancement Owner's Manual: Model R1D, 1978.
of noisy images. IEEE Transactions on Systems, Man and Cy- [74] Telesensory Systems, Inc., Palo Alto, California. Optacon
bernetics, SMC-7(6), 1976. Announcement, Available from http://www.telesensory.com,
[46] C.A. Lindley. Practical Image Processing in C. John Wiley and 1996.
Sons, Inc., New York, 1991. [75] J.A. Terry and H. Hsiao. Tactile feedback in a computer mouse.
[47] J.M. Loomis. On the tangibility of letters and braille. Perception Proceedings of the 14th Northeast Conference on Bioengineer-
and Psychophysics, 29, 1981. ing, 1988.
[48] J.M. Loomis. Tactile pattern perception. Perception, 10, 1981.
[49] J.M. Loomis and S.J. Lederman. Tactual perception. In K.R. [76] J.P. Thomas. JAWS User's Guide and Reference Manual, Sec-
Bo, L. Kaufman, and J.P. Thomas, editors, Handbook of Per- ond Edition. Henter-Joyce, Inc., St. Petersburg, FL, 1994.
ception and Human Performance. John Wiley and Sons, Inc., [77] C.M. Thompson and L. Shure. Image Processing Toolbox: For
1986. Use with MATLAB. The Math Works, Inc., Natick, Mas-
[50] B. Lowenfeld. Eects of blindness on the cognitive functions of sachusetts, 1995.
children. In B. Lowenfeld, editor, Berthold Lowefeld on Blind- [78] J.H. Todd. Resources, media, and technology. In G.T. Scholl, ed-
ness and Blind People. American Foundation for the Blind, New itor, Foundations of Education for the Blind and Visually Hand-
York, 1981. icapped Children and Youth: Theory and Practice. American
[51] Matsumoto Kosan Co. LTD. Stereo copying system for the blind. Foundation for the Blind, 1986.
Product handbook, 1990. [79] J.T. Tou and R.C. Gonzalez. Pattern Recognition Principles.
[52] T. Massie and K. Salisbury. The PHANToM haptic interface: Addison-Wesley Publishing Company, Reading, Massachusetts,
a device for probing virtual objects. In Symposium on Haptic 1974.
Interfaces for Virtual Environment and Teleoperator Systems, [80] B. Tversky and D. Baratz. Memory for faces: Are caricatures
ASME Winter Annual Meeting, 1994. better than photographs. Memory and Cognition, 13(1), 1985.
WAY AND BARNER: AUTOMATIC VISUAL TO TACTILE TRANSLATION 13

[81] G.C. Vanderheiden. Systems 3 - an interface to graphic com- Thomas Way was born in Rockville, Mary-
puters for blind users. Proceedings of the RESNA 13th Annual land, in 1962. He received the B.A. degree in
Conference, 1990. Film and Television Production from the Uni-
[82] G.C. Vanderheiden. Dynamic and static strategies for nonvisual versity of Maryland at College Park in 1984,
presentation of graphic information. Electronic Proceedings of and spent the following nine years in Holly-
the 1994 EASI High Resolution Tactile Graphics Conference, wood, California working in the entertainment
Available from http://www.rit.edu/easi/, 1994. industry as a writer, producer, director and ra-
[83] G.C. Vanderheiden. The audiotactile snapshot approach to pro- dio announcer. He received the M.S. in Com-
viding access to graphic images as they are encountered: an ex- puter and Information Sciences from the Uni-
perimental project at the trace center, university of wisconsin. versity of Delaware in 1996, and is currently
White paper, available from http://trace.wisc.edu., 1996. pursuing Ph.D. studies there. His research in-
[84] M.E. Ward. The visual system. In G.T. Scholl, editor, Foun- terests include human-computer interaction, image processing, com-
dations of Education for the Blind and Visually Handicapped piler optimization, and high performance computing. He is married
Children and Youth: Theory and Practice. American Founda- and has a daughter.
tion for the Blind, 1986.
[85] T. Way and K. Barner. Towards automatic generation of tactile
graphics. Proceedings of the RESNA '96 Annual Conference,
1996.
[86] T.P. Way. Automatic generation of tactile graphics. Master's Kenneth Barner was born in Montclair, New
thesis, University of Delaware, 1996. Jersey, in 1963. He received a B.S.E.E. de-
[87] S. Weinstein. Intensive and extensive aspects of tactile sensi- gree (magna cum laude ) from Lehigh Univer-
tivity as a function of body part, sex, and laterality. In D.R. sity, Bethlehem, Pennsylvania, in 1987. He re-
Kenshalo, editor, The Skin Senses. Charles C. Thomas, Spring- ceived the M.S.E.E. and Ph.D. degrees from
eld, IL, 1968. the University of Delaware, Newark, Delaware,
[88] B.W. White, F.A. Saunders, L. Scadden, P. Bach-y Rita, and in 1989 and 1990, respectively. Dr. Barner
C.C. Collins. Seeing with skin. Perception and Psychophysics, was the duPont Teaching Fellow and a Visit-
7, 1970. ing Lecturer at the University of Delaware in
[89] S.F. Wiker, G. Vanderheiden, S. Lee, and S. Arndt. Development 1991 and 1992, respectively. He is currently a
of tactile mice for blind access to computers: Importance of Research Engineer at the Applied Science and
stimulation locus, object size and vibrotactile display resolution. Engineering Laboratories of the A. I. duPont Institute/University of
Proceedings of the Human Factors Society 35th Annual Meeting, Delaware, and an Assistant Research Professor in the Department of
1991. Electrical Engineering at the University of Delaware. His research in-
[90] D.H. Willis. Relationship between visual acuity, reading mode, terests include signal and image processing, nonlinear systems, speech
and school systems for blind children: A 1979 replication. Amer- enhancement, recognition, and synthesis, and pattern recognition. He
ican Printing House, 1979. is a member of Tau Beta Pi, Eta Kappa Nu, and Phi Sigma Kappa.
14 IEEE TRANSACTIONS ON REHABILITATION ENGINEERING

TABLE I
Summary of information bandwidth limitations for three senses [39].

Sense Modality Limit bits=sec

Skin (vibrotactile) 102
Ear 104
Eye 106

TABLE II
Reading modes used by a group of 7,987 totally blind students.

Method Percentage 5
Aural 61
Braille 37
Braille & Large Type 1
Large Type 1

TABLE III
Summary of parameters relevant to TACTICS and tactile image perception.

Factor Parameters
Ratio of tactual to visual bandwidths 1:10000
Minimum discernible separation of two points 2.5mm
(static)
Minimum discernible displacement of a point 0.002mm
on a smooth surface
Height of braille dot 0.2-0.5mm
Minimum discernible separation of groves in 1.0mm
grating (dynamic)
Resolution of laser printer 76.2-152.4 dots/mm (300-600 dpi)
Resolution of microcapsule paper (expanded) 1-5 capsules/mm
Expanded displacement of microcapsule paper 0.2-1.0mm
Resolution of human ngertip 1 dot/mm
Resolution of ngertip compares with: very blurry vision
Human memory organization Hierarchical: general to specic
Congenital blindness onset up to age 5
Adventitious blindness onset after age 5
Blind population (worldwide) 30-40 million
Blind population (U.S.) 500,000
Braille uency (U.S. blind population) <16%
Best size for tactile image 7.62-12.7cm (3-5in) on a side
WAY AND BARNER: AUTOMATIC VISUAL TO TACTILE TRANSLATION 15

Microcapsules

Fig. 1. Microcapsule paper showing layer of polystyrene microcapsules on polyethylene or paper transport medium.

Image

Fig. 2. Microcapsule paper after image is axed to the surface by photocopying or ink drawing.

Heating element

Paper path

Transport
rollers

Fig. 3. Simplied view of the Tactile Image Enhancer, showing internal workings of the device for expanding previously exposed capsule
paper.

Expanded capsules

Fig. 4. Microcapsule paper after exposure in image enhancer, showing expanded capsules.

S Denotes active or
vibrating pins

(a) Use of Optacon. User places index n- (b) Active pin matrix display of the
ger of one hand on vibrotactile pin array Optacon, demonstrating display of
and guides scanner across material to be the capital letter S.
viewed with other hand.
Fig. 5. Optacon
16 IEEE TRANSACTIONS ON REHABILITATION ENGINEERING

(a) Unprocessed (b) Sobel edge detection (c) Blurring

(d) K-means (e) Adaptive K-means (f) Negation

Fig. 6. Image processing algorithms

(a) Before (b) After

Fig. 7. Median ltering of a noisy image