A Generative Vision Model That Trains With High Data Efficiency and Breaks Text-Based Captchas

RESEARCH ARTICLES
Cite as: D. George et al., Science

10.1126/science.aag2612 (2017).
A generative vision model that trains with high data

efficiency and breaks text-based CAPTCHAs
D. George,* W. Lehrach, K. Kansky, M. Lzaro-Gredilla,* C. Laan, B. Marthi, X. Lou, Z. Meng, Y. Liu, H. Wang,
A. Lavin, D. S. Phoenix
Vicarious AI, 2 Union Square, Union City, CA 94587, USA.
*Corresponding author. Email: [email protected] (D.G.); [email protected] (M.L.-G.)
Learning from few examples and generalizing to dramatically different situations are capabilities of
human visual intelligence that are yet to be matched by leading machine learning models. By drawing
inspiration from systems neuroscience, we introduce a probabilistic generative model for vision in which
message-passing based inference handles recognition, segmentation and reasoning in a unified way. The
Downloaded from http://science.sciencemag.org/ on October 26, 2017

model demonstrates excellent generalization and occlusion-reasoning capabilities, and outperforms deep
neural networks on a challenging scene text recognition benchmark while being 300-fold more data
efficient. In addition, the model fundamentally breaks the defense of modern text-based CAPTCHAs by
generatively segmenting characters without CAPTCHA-specific heuristics. Our model emphasizes aspects
like data efficiency and compositionality that may be important in the path toward general artificial
intelligence.
The ability to learn and generalize from a few examples is a when partially transparent objects occupy the same spatial
hallmark of human intelligence (1). CAPTCHAs, images used locations (1316). Lateral connections in the visual cortex are
by websites to block automated interactions, are examples of implicated in enforcing contour continuity (17, 18). Contours
problems that are easy for humans but difficult for comput- and surfaces are represented using separate mechanisms that
ers. CAPTCHAs are hard for algorithms because they add interact (1921), enabling the recognition and imagination of
clutter and crowd letters together to create a chicken-and-egg objects with unusual appearance for example a chair made
problem for character classifiers the classifiers work well of ice. The timing and topography of cortical activations give
for characters that have been segmented out, but segmenting clues about contour-surface representations and inference al-
the individual characters requires an understanding of the gorithms (22, 23). These insights based on cortical function
characters, each of which might be rendered in a combinato- are yet to be incorporated into leading machine learning
rial number of ways (25). A recent deep-learning approach models.
for parsing one specific CAPTCHA style required millions of We introduce a hierarchical model called the Recursive
labeled examples from it (6), and earlier approaches mostly Cortical Network (RCN) that incorporates these neuroscience
relied on hand-crafted style-specific heuristics to segment out insights in a structured probabilistic generative model frame-
the character (3, 7); whereas humans can solve new styles work (5, 2427).
without explicit training (Fig. 1A). The wide variety of ways In addition to developing RCN and its learning and infer-
in which letterforms could be rendered and still be under- ence algorithms, we applied the model to a variety of visual
stood by people is illustrated in Fig. 1. cognition tasks that required generalizing from one or a few
Building models that generalize well beyond their train- training examples: parsing of CAPTCHAs, one-shot and few-
ing distribution is an important step toward the flexibility shot recognition and generation of handwritten digits, occlu-
Douglas Hofstadter envisioned when he said that for any sion reasoning, and scene text recognition. We then com-
program to handle letterforms with the flexibility that human pared its performance to state of the art models.
beings do, it would have to possess full-scale artificial intelli-
gence (8). Many researchers have conjectured that this could Recursive cortical network
be achieved by incorporating the inductive biases of the vis- RCN builds on existing compositional models (24, 2832) in
ual cortex (912), utilizing the wealth of data generated by important ways [section 6 of (33)]. Although grammar based
neuroscience and cognitive science research. In the mamma- models (24) have the advantage of being based on well-
lian brain, feedback connections in the visual cortex play known ideas from linguistics, they either limit interpreta-
roles in figure-ground-segmentation, and in object-based top- tions to single trees or are computationally infeasible when
down attention that isolates the contours of an object even using attributed relations (32). The seminal work on AND-OR
First release: 26 October 2017 www.sciencemag.org (Page numbers not final at time of first release) 1
templates and tree-structured compositional models (34) has representations in detail.
the advantage of simplified inference, but is lacking in selec- Figure 2B shows two subnetworks (black and blue) within
tivity owing to the absence of lateral constraints (35). Models a level of the RCN contour hierarchy. The filled and empty
from another important class (25, 29) use lateral constraints, circular nodes in the graph are binary random variables that
but rather than gradually building invariance through a pool- correspond to features and pools respectively. Each feature
ing structure (36), they use parametric transformations for node encodes an AND relation of its child pools, and each
complete scale, rotation and translation invariance at each pool variable encodes the OR of its child features, similar to
level. Custom inference algorithms are required, but those AND-OR graphs (34). Lateral constraints, represented as rec-
are not effective in propagating the effect of lateral con- tangular factor nodes, coordinate the choices between the
straints beyond local interactions. The representation of con- pools they connect to. The two subnetworks, which can cor-
tours and surfaces in (37) do not model their interactions, respond to two objects or object parts, share lower level-fea-
choosing instead to model these as independent mechanisms. tures.
RCNs and Composition Machines (CM) (32) share the moti- Figure 2C shows a three-level network that represents the
vation of placing compositional model ideas in a graphical contours of a square. The features at the lowest, intermediate

model formulation. However, CMs representational choice of and top levels respectively represent line segments, corners
composed distributions using a single layer of random and the entire square. Each pool variable pools over different
variables to collapse feature-detection, pooling and lateral co- deformations, small translations, scale changes etc., of a cen-
ordination leads to an expanded state space, which in turn tered feature, thus introducing the corresponding invari-
constrains the model to a greedy inference and parsing pro- ances. Without the lateral connections between the pools (the
cess. In general, because of the varied and conflicting repre- gray squares in Fig. 2C), generating from a feature node rep-
sentational choices, inference in compositional models has resenting a corner can create misaligned line segments, as
relied on custom-crafted methods for different model instan- shown in Fig. 3A. Lateral connections between the pools pro-
tiations, including solving stochastic partial differential equa- vide selectivity (35) by ensuring that the choice of a feature
tions (30), sampling based algorithms (24), and pruned in one pool affects the choice of features in pools it is con-
dynamic programming (29). nected to, creating samples where the contours vary more
RCN integrates and builds upon various ideas from com- smoothly. The flexibility of lateral constraints is controlled
positional models hierarchical composition, gradual build- through perturb-factor, a hyperparameter that is specified
ing of invariances, lateral connections for selectivity, contour- per level. Through multiple layers of feature pooling, lateral
surface factorization and joint-explanation based parsing connections, and compositions, a feature node at the top level
into a structured probabilistic graphical model such that Be- comes to represent an object that can be recognized with
lief Propagation (38) can be used as the primary approximate some level of translation, scale and deformation invariance.
inference engine [section 6 of (33)]. Experimental neurosci- Multiple objects are represented in the same shape hier-
ence data provided important guidance on the representa- archy by sharing their parts (Fig. 2B). When multiple parents
tional choices [section 7 of (33)], which were then confirmed converge on a single child feature (feature node e in Fig.
to be beneficial using experimental studies. We now discuss 2B), this will be active when any parent is active (OR-gate in
the representation of RCN and its inference and learning al- the graphical model), and the child feature is allowed to be
gorithms. Mathematical details are discussed in sections 2 to part of both parents if evidence allows, unlike the exclusive
5 of (33). sharing in AND-OR graph grammars (24). Even when two
higher-level features share some of the same lower-level fea-
Representation tures and pools, the higher-level features lateral networks are
In RCN, objects are modeled as a combination of contours kept separate by making copies of the lower-level feature for
and surfaces (Fig. 2A). Contours appear at the boundaries of each specific higher-level feature it participates in, as shown
surfaces, both at the outline of objects and at the border be- in Fig. 2B. Parent-specific copies of lateral networks serve to
tween the surfaces that compose the object. Surfaces are achieve higher-order interactions compared to pairwise con-
modeled using a Conditional Random Field (CRF) which cap- nections, similar to the state copying mechanism used in
tures the smoothness of variations of surface properties. Con- higher-order networks (40). This was also found to be im-
tours are modeled using a compositional hierarchy of portant for message-passing to achieve accurate results and
features (28, 39). Factored representation of contours (shape) is reminiscent of techniques used in dual decomposition (41).
and surfaces (appearance) enables the model to recognize ob- Hierarchy in the RCN network plays two roles. First, it ena-
ject shapes with dramatically different appearances without bles the representation of deformations gradually through
training exhaustively on every possible shape and appearance multiple levels, spreading the amount of variation across lay-
combination. We now describe the shape and appearance ers (Fig. 3B). Second, hierarchy provides efficiency through
the sharing of features between different objects (42). Both of mask on the input image. By considering combinations of ob-
these result in efficient learning and inference through ject hypotheses, i.e., parses, that produce spatially contiguous
shared computations. masks when their 2d-masks overlap, we create a topological
Surfaces are modeled using a pairwise CRF (Fig. 3C). Lo- ordering of the parses by sorting them according to masks
cal surface patch properties like color, texture, or surface nor- that are contained in other masks. This results in a recursive
mal are represented by categorical variables, whose computation of the score where only a linear number of can-
smoothness of variation is enforced by the lateral factors didate parses need to be evaluated in searching for the best
(gray squares in Fig. 2). Contours generated by the contour- parse. See section 4.7 of (33) for more details.
hierarchy interact with the surface CRF in a specific way: con-
tours signal the breaks in continuity of surfaces that occur Learning
both within an object and between the object and its back- Features and lateral connections up to the penultimate level
ground, a representational choice inspired by neurobiology of the network are trained unsupervised using a generic 3D
(19). Figure 3, B and D, shows samples generated from an object data set that is task agnostic and rendered only as con-
RCN. tour images. The resulting learned features vary from simple

line segments at the lower levels to curves and corners at the
Inference higher levels.
In order to parse a scene, RCN maintains hierarchical graphs Consider a partially learned model, where new features
for multiple object instances at multiple locations tiling the are being learned at level k, where features up to level k-1
scene. The parse of a scene can be obtained via maximum a have already been learned and finalized, and a few features
posteriori (MAP) inference on this complex graph, which re- have been learned at level k (Fig. 4B). When a training image
covers the best joint configuration including object identities is presented, the first step is to find a MAP explanation for
and their segmentations [section 4 of (33)]. Although the the contours of that image using the existing features at level
RCN network is extremely loopy, we found that message- k. This is identical to the inference problem described earlier
passing (38), with a schedule that is inspired by the timing of of finding the MAP solution for a scene. The contours that
activations in the visual cortex (9, 20), resulted in fast and remain unexplained are parsed using features at level k-1, and
accurate inference. An input image is first passed through new features are proposed from their contour-continuous
PreProc, which converts pixel values to edge likelihoods using conjunctions. Repeating this process for all the training im-
a bank of Gabor-like filters. Partial assignments that corre- ages accumulates counts on the usage of different features at
spond to object hypotheses are then identified using a for- level k, and the final features for this level are selected by
ward and backward message passing in the network, and a optimizing an objective function that balances compression
complete approximate MAP solution is found by solving the and reconstruction error (31). The same process is repeated
scene-parsing problem on the graph of object hypotheses level-by-level [see section 5.1 of (33)].
(Fig. 4). The forward pass gives an upper-bound on the log- The lateral graph structure, which specifies the connectiv-
probability of the nodes at the top level. The backward pass ity between pool pairs, is learned from the contour connec-
visits the high-scoring forward-pass hypotheses one by one, tivity of input images. At the first pooling level, pools with
in a manner similar to a top-down attention process (43, 44), features that are adjacent in the input contours are connected
running a conditional inference that assumes that all other with each other. This process is repeated recursively in the
nodes are off to find an approximate MAP configuration for hierarchy where lateral connections at the higher levels are
the object (Fig. 4A). The backward pass can reject many ob- inferred from adjacency in the lower-level graphs.
ject hypotheses that were falsely identified in the forward Features at the topmost level represent whole objects.
pass. These are obtained by finding the MAP configuration of a
The global MAP configuration is a subset of all the object new object up to the penultimate level of the network, con-
hypotheses generated from the forward and backward passes. necting pool pairs at the penultimate level according to the
The number of objects in the scene is inferred as part of this contour continuity of the input object, and then storing the
MAP solution. In addition to searching over an exponentially conjunction of activations at the penultimate level as a fea-
large number of subsets, finding the global MAP requires rea- ture in the top-most level. See section 5 of (33) for details.
soning about high-order interactions between different hy- Once the set of lower-level features and lateral connec-
potheses. We developed an approximate dynamic tions are trained, they can be used for different domains by
programming (DP) method that solves this in linear time. The tuning a few hyper-parameters [section 8.3 of (33)]. The filter
DP algorithm exploits the fact that each object hypothesis oc- scales in the PreProc are chosen depending on the image and
cupies a contiguous region that can be represented as a 2d object size, and the flexibility of the lateral connections is set
to match the distortions in the data. In addition, the features
at the lowest level have a smoothing parameter that sets an the network deteriorates rapidly with even minor perturba-
estimate on the probability that an edge pixel is ON owing to tions to the spacing of characters that are barely perceptible
noise. This parameter can be set according to the noise levels to humans 15% more spacing reduced accuracy to 38.4%,
in a domain. and 25% more spacing reduced accuracy to just 7%. This sug-
gests that the deep-learning method learned to exploit the
Results specifics of a particular CAPTCHA rather than learning mod-
A CAPTCHA is considered broken if it can be automatically els of characters that are then used for parsing the scene. For
solved at a rate above 1% (3). RCN was effective in breaking RCN, increasing the spacing of the characters results in an
a wide variety of text-based CAPTCHAs with very little train- improvement in the recognition accuracy (Fig. 5B).
ing data, and without using CAPTCHA-specific heuristics The wide variety of character appearances in BotDetect
(Fig. 5). It was able to solve reCAPTCHAs at an accuracy rate (Fig. 5C) demonstrates why the factorization of contours and
of 66.6% (character level accuracy of 94.3%), BotDetect at surfaces is important: models without this factorization
64.4%, Yahoo at 57.4% and PayPal at 57.1%, significantly could latch on to the specific appearance details of a font,
above the 1% rate at which CAPTCHAs are considered inef- thereby limiting their generalization. The RCN results are

fective (3). The only differences in architecture across differ- based on testing on 10 different styles of CAPTCHAs from
ent CAPTCHA tasks are the sets of clean fonts used for BotDetect, all parsed based on a single network trained on 24
training and the different choices of a few hyper-parameters, training example per character, and using the same parsing
which depend on the size of the CAPTCHA image and the parameters across all styles. Although BotDetect CAPTCHAs
amount of clutter and deformations. These parameters are can be parsed using contour information alone, using the ap-
straightforward to set by hand, or can be tuned automatically pearance information boosted the accuracy from 61.8% to
via cross validation on an annotated CAPTCHA set. Noisy, 64.4%, using the same appearance model across all data sets.
cluttered and deformed examples from the CAPTCHAs were See section 8.4.6 of (33) for more details.
not used for training, yet RCN was effective in generalizing RCN outperformed other models on one-shot and few-
to those variations. shot classification tasks on the standard MNIST handwritten
For reCAPTCHA parsing at 66.6% accuracy, RCN required digit data set [section 8.7 of (33)]. We compared RCNs clas-
only five clean training examples per character. The model sification performance on MNIST as we varied the number of
uses three parameters that affect how single characters are training examples from 1 to 100 per category. CNN compari-
combined together to read out a string of characters, and sons were made with two state-of-the art models, a LeNet-5
these parameters were both independent of the length of the (45) and the VGG-fc6 CNN (46) with its levels pre-trained for
CAPTCHAs and were robust to the spacing of the characters ImageNet (47) classification using millions of images. The
[Fig. 5B and section 8.4 of (33)]. In addition to obtaining a fully-connected-layer fc6 of VGG-CNN was chosen for com-
transcription of the CAPTCHA, the model also provides a parison because it gave the best results for this task compared
highly accurate segmentation into individual characters, as to other pre-trained levels of the VGG-CNN, and compared to
shown in Fig. 5A. To compare, human accuracy on reCAP- other pre-trained CNNs that used the same data set and edge
TCHA is 87.4%. Because many input images have multiple pre-processing as RCN [section 5.1 of (33)]. In addition, we
valid interpretations (Fig. 5A), parses from two humans agree compared against the Compositional Patch Model (48) that
only 81% of the time. recently reported state-of-the-art performance on this task.
In comparison to RCNs, a state-of-the-art CNN (6) re- RCN outperformed the CNNs and the CPM (Fig. 6A). The one-
quired a 50,000-fold larger training set of actual CAPTCHA shot recognition performance of RCN was 76.6% vs 68.9% for
strings, and it was less robust to perturbations to the input. CPM and 54.2% for VGG-fc6. RCN was also robust to different
Because the CNN required a large number of labeled exam- forms of clutter that were introduced during testing, without
ples, this control study used a CAPTCHA-generator that we having to expose the network to those transformations dur-
created to emulate the appearance of reCAPTCHAs [see sec- ing training. In comparison, such out-of-sample test exam-
tion 8.4.3 of (33)]. The approach used a bank of position-spe- ples had a large detrimental effect on the generalization
cific CNNs, each trained to discriminate the letter at a performance of CNNs (Fig. 6B). To isolate the contributions
particular position. Training the CNNs to achieve a word-ac- of lateral connections, forward pass, and backward pass to
curacy rate of 89.9% required over 2.3 million unique train- RCNs accuracy, we conducted lesion studies that selectively
ing images, created using translated crops for data turned off these mechanisms. The results, summarized in Fig.
augmentation, from 79,000 distinct CAPTCHA words. The re- 6C, show that all these mechanisms contribute significantly
sulting network fails on string lengths not present during toward the performance of RCNs. RCN networks with two
training, and more importantly, the recognition accuracy of levels of feature detection and pooling were sufficient to get
the best accuracy performance on character parsing tasks.
The effect of increasing the number of levels in the hierarchy segmentation of the characters (Fig. 7E) that the competing
is to reduce the inference time as detailed in section 8.11 of methods do not provide.
(33).
As a generative model, RCN outperformed Variational Discussion
Auto Encoders (VAE) (49) and DRAW (50) on reconstructing Segmentation resistance, the primary defense of text-based
corrupted MNIST images (Fig. 7, A and B). DRAWs ad- CAPTCHAs, has been a general principle that enabled their
vantage over RCN for the clean test set is not surprising be- automated generation (2, 3). Although specific CAPTCHAs
cause DRAW is learning an overly flexible model that almost have been broken before using style-specific segmentation
copies the input image in the reconstruction, which hurts its heuristics (3, 7), those attacks could be foiled easily by minor
performance on more cluttered data sets [section 8.9 of (33)]. alterations to CAPTCHAs. RCN breaks the segmentation de-
On the Omniglot data (1), examples generated from RCN after fense in a fundamental way and with very little training data,
one-shot training showed significant variations, while still which suggests that websites should move to more robust
being identifiable as the original category [Fig. 7D and sec- mechanisms for blocking bots.
tion 8.6 of (33)]. Compositional models have been successfully used in the

To test occlusion reasoning (5153) we created a variant past for generic object recognition and scene parsing, and our
of the MNIST data set by adding a rectangle to each valida- preliminary experiments [section 8.12 of (33)] indicate that
tion/test image such that some parts of the digit were oc- RCN could be applicable in those domains as well (Fig. 8).
cluded by the rectangle and some parts of the rectangle were The RCN formulation opens up compositional models to a
occluded by the digit [Fig. 7C and section 8.8 of (33)]. Occlu- wider array of advanced inference and learning algorithms
sion relationships in these images cannot be deduced as a developed in graphical models, potentially leading to im-
simple layering of one object in front of the other. Classifica- provements that build on their prior successes in real-world
tion on this data set is challenging because many parts of the scene parsing (56, 57). Despite the advantage of being a gen-
digit are occluded by the rectangle, and because the rectangle erative model, RCN needs several improvements to achieve
acts as clutter. If the rectangle is detected and segmented out, superior performance on ImageNet-scale (47) data sets. Flex-
its effect on the evidence for a particular digit can be ex- ible merging of multiple instances, the use of surface appear-
plained away using the RCN generative model, thereby im- ance at all levels of the hierarchy during forward and
proving the accuracy of classification and segmentation. RCN backward inference, more sophisticated pooling structures
was tested for classification accuracy and for occlusion rea- that learn to pool over 3D transformations, and generative
soning on this challenging data set. Classification accuracy modeling of scene context and background are problems that
without explaining away was 47.0%. Explaining away the rec- need to be investigated and integrated with RCN [section 8.13
tangle boosts the classification accuracy to 80.7%. In addi- of (33)].
tion, RCN was used to parse the scene by reasoning about the The high data efficiency of RCN, compared to whole-im-
occlusion relation between the rectangle and the digit. The age models like CNNs and VAEs, derives from the fact that
model was successful at predicting the precise occlusion re- RCN encodes strong assumptions in its structure. Recent
lations of the test image as shown in Fig. 7C, obtaining a neural networks models incorporate ideas of compositional-
mean intersection over union (IOU) of 0.353 measured over ity using a spatial attention window (58), but their current
the occluded regions. instantiations need good separation between the objects in
Last, RCN was tested on the ICDAR-13 Robust Reading an uncluttered setting because each attention window is
data set (54), a benchmark for text recognition in real world modeled using a whole-image VAE. Incorporation of RCNs
images (Fig. 7E). For this test, we enhanced the parsing algo- object and part-based compositionality into neural network
rithm to include prior knowledge about n-gram and word sta- models would be an interesting research direction. Unlike
tistics, and about geometric priors related to the layout of neural networks, the current version of RCN learning algo-
letters in a scene, which includes spacing, relative sizes and rithms need clean training data, a drawback we intend to ad-
appearance consistency [see section 8.5 of (33)]. We com- dress using gradient based learning as well as message-
pared our result against top participants of the ICDAR com- passing based approaches (59).
petition, and against a recent deep learning approach (55) Combining RCN with Bayesian Program Learning (BPL)
(Table 1). The RCN model outperformed the top contender, (1) is another avenue for future investigations. BPL has the
PhotoOCR, by 1.9%, despite PhotoOCR using 7.9 million advantage of precisely modeling the sequential causal mech-
training images, whereas RCN used 1,406 training images se- anisms, e.g., the stroke generation in the Omniglot data set,
lected using model-based clustering from 25,584 font images. but its inference depends on the contours being separated
RCN achieved better accuracy on this task while being 300 from the background something RCN can easily provide.
times more data efficient, in addition to providing a detailed More generally, BPL and RCN-like graphical models could be
combined to obtain the expressive power and efficient infer- from Google Fonts, which resulted in 25584 character train-
ence required to model the parallel and sequential processes ing images. From this we selected a set of training images
(60) involved in perception and cognition. using an automated greedy font selection approach. We ren-
Of course, Douglas Hofstadters challenge understand- dered binary images for all fonts and then used the resulting
ing letterforms with the same efficiency and flexibility of hu- images of the same letter to train an RCN. This RCN is then
mans still stands as a grand goal for artificial intelligence. used to recognize the exact images it was trained on, provid-
People use a lot more commonsense knowledge, in context- ing a compatibility score (between 0.0 and 1.0) for all pairs of
sensitive and dynamic ways, when they identify letterforms fonts of the same letter. Finally, using a threshold (=0.8) as
(Fig. 1C, iii). Our works suggests that incorporating inductive the stopping criterion, we greedily select the most representa-
biases from systems neuroscience can lead to robust, gener- tive fonts until 90% of all fonts are represented, which re-
alizable machine-learning models that demonstrate high sulted in 776 unique training images. The parser is trained
data efficiency. We hope that this work inspires improved using 630 word images and the character ngrams are trained
models of cortical circuits (61, 62) and investigations that using words from the Wikipedia.
combine the power of neural networks and structured prob- RCN classification experiments on the MNIST data set are

abilistic models toward general artificial intelligence systems. done by up-sampling the images by a factor of 4. For each
training setting, two pooling hyperparameters of the model
Methods summary were adapted using an independent validation set of rotated
For reCAPTCHA experiments, we downloaded 5500 reCAP- MNIST digits. Several ways of pre-training the CNN are ex-
TCHA images from google.com reCAPTCHA page, of which plored as part of the baselines. To understand the perfor-
500 were used as validation set for parameter tuning, and ac- mance of the networks on noisy MNIST data, we created six
curacy numbers are reported on the remaining 5000. The im- variants of noise, each one with three levels of severity. For
ages were scaled up by a factor of 2. A similar-looking font to occlusion reasoning, the RCN network was trained with 11
those used in reCAPTCHA, Georgia, was identified by visual categories: ten MNIST digit categories with 20 examples for
comparison from the fonts available on the local system. RCN category and the rectangular ring category with one example.
was trained on a few rotations of the lowercase and upper- Reconstruction experiments on the MNIST data set used net-
case characters from this font. Hyperparameters were opti- works that were trained only on clean MNIST images which
mized using the validation set. Human accuracy on the were then tested for mean squared reconstruction error on 6
reCAPTCHA data set was estimated using Amazon Mechani- different noise variants, each with 3 levels of severity. Full
cal Turk (AMT) using U.S. based workers. methods are available in supplemental materials.
Emulated reCAPTCHA data sets, used to train the neural REFERENCES AND NOTES
network for control experiments, were created using Image- 1. B. M. Lake, R. Salakhutdinov, J. B. Tenenbaum, Human-level concept learning
Magick to produce distortions that are qualitatively similar through probabilistic program induction. Science 350, 13321338 (2015).
to the original reCAPTCHA. The emulated data generator is doi:10.1126/science.aab3050 Medline
2. K. Chellapilla, P. Simard, Using machine learning to break visual human interaction
used as an unlimited source to generate random batches for proofs (HIPs), in Advances in Neural Information Processing Systems 17) (2004)
training the neural network. Neural network optimization pp. 265272.
was run for 80 epochs, where data are permuted at the start 3. E. Bursztein, M. Martin, J. C. Mitchell, Text-based CAPTCHA strengths and
of every epoch; data were also augmented by random trans- weaknesses, in Proceedings of the 18th ACM Conference on Computer and
Communications Security (ACM, 2011), pp. 125138.
lations of up to 5 pixels in each cardinal direction per epoch. 4. G. Mori, J. Malik, Recognizing objects in adversarial clutter: Breaking a visual
Similar methods were used for BotDetect, PayPal and Ya- CAPTCHA, in 2003 IEEE Conference on Computer Vision and Pattern Recognition
hoo CAPTCHAs. For BotDetect, we downloaded a data set of (IEEE Computer Society, 2003), pp. I-134I-141.
50-100 images per CAPTCHA style for determining the pars- 5. V. Mansinghka, T. D. Kulkarni, Y. N. Perov, J. Tenenbaum, Approximate bayesian
image interpretation using generative probabilistic graphics programs, in
ing parameters and training setup, and another 100 images Advances in Neural Information Processing Systems 26 (2013), pages 15201528.
as a testing data set on which the network is not tuned. As 6. I. Goodfellow, Y. Bulatov, J. Ibarz, S. Arnoud, V. Shet, Multi-digit number
training images for the system, we selected a series of fonts recognition from street view imagery using deep convolutional neural networks,
and scales from those available on the system by visually paper presented at the International Conference on Learning Representations
(ICLR) 2014, Banff, Canada, 14 to 16 April 2014.
comparing a few examples of the BotDetect CAPTCHAs. The 7. E. Bursztein, J. Aigrain, A. Moscicki, J. C. Mitchell, The End is nigh: Generic Solving
BotDetect test images were rescaled by a factor of 1.45. Pars- of text-based CAPTCHAs, paper presented at the 8th USENIX Workshop on
ing parameters were optimized using the validation set, and Offensive Technologies (WOOT 14), San Diego, CA, 19 August 2014.
the transferability of the parsing parameters were tested by 8. D. R. Hofstadter, Metamagical Themas: Questing for the Essence of Mind and Pattern
(Basic Books, 1985).
adapting the parameters for each style separately and then 9. T. S. Lee, D. Mumford, Hierarchical Bayesian inference in the visual cortex. JOSA A
testing those parameters on the other styles. 20, 14341448 (2003). doi:10.1364/JOSAA.20.001434 Medline
For training RCN to parse ICDAR, we obtained 492 fonts 10. T. L. Griffiths, N. Chater, C. Kemp, A. Perfors, J. B. Tenenbaum, Probabilistic
models of cognition: Exploring representations and inductive biases. Trends Cogn. 100, 212224 (2006). doi:10.1016/j.jphysparis.2007.01.001 Medline
Sci. 14, 357364 (2010). doi:10.1016/j.tics.2010.05.004 Medline 36. T. Serre, L. Wolf, S. Bileschi, M. Riesenhuber, T. Poggio, Robust object recognition
11. T. S. Lee, The visual systems internal model of the world, in Proceedings of the with cortex-like mechanisms. IEEE Trans. Pattern Anal. Mach. Intell. 29, 411426
IEEE, vol. 103 (IEEE, 2015), pp. 13591378 (2007). doi:10.1109/TPAMI.2007.56 Medline
12. D. Kersten, A. Yuille, Bayesian models of object perception. Curr. Opin. Neurobiol. 37. C. Guo, S.-C. Zhu, Y. N. Wu, Primal sketch: Integrating structure and texture.
13, 150158 (2003). doi:10.1016/S0959-4388(03)00042-4 Medline Comput. Vis. Image Underst. 106, 519 (2007). doi:10.1016/j.cviu.2005.09.004
13. C. D. Gilbert, W. Li, Top-down influences on visual processing. Nat. Rev. Neurosci. 38. J. Pearl, Probabilistic Reasoning in Intelligent Systems: Networks of Plausible
14, 350363 (2013). doi:10.1038/nrn3476 Medline Inference (Morgan Kaufmann, 1988).
14. V. A. F. Lamme, P. R. Roelfsema, The distinct modes of vision offered by 39. T. Wu, S.-C. Zhu, A numerical study of the bottom-up and top-down inference
feedforward and recurrent processing. Trends Neurosci. 23, 571579 (2000). processes in and-or graphs. Int. J. Comput. Vis. 93, 226252 (2011).
doi:10.1016/S0166-2236(00)01657-X Medline doi:10.1007/s11263-010-0346-6
15. P. R. Roelfsema, V. A. F. Lamme, H. Spekreijse, Object-based attention in the 40. J. Xu, T. L. Wickramarathne, N. V. Chawla, Representing higher-order
primary visual cortex of the macaque monkey. Nature 395, 376381 (1998). dependencies in networks. Sci. Adv. 2, e1600028 (2016).
doi:10.1038/26475 Medline doi:10.1126/sciadv.1600028 Medline
16. E. H. Cohen, F. Tong, Neural mechanisms of object-based attention. Cereb. Cortex 41. D. Sontag, A. Globerson, T. Jaakkola, Introduction to dual decomposition for
25, 10801092 (2015). doi:10.1093/cercor/bht303 Medline inference, in Optimization for Machine Learning (MIT Press, 2010), pp. 137.
17. D. J. Field, A. Hayes, R. F. Hess, Contour integration by the human visual system: 42. E. Bienenstock, S. Geman, D. Potter, Compositionality, MDL priors, and object
Evidence for a local association field. Vision Res. 33, 173193 (1993). recognition, in Advances in Neural Information Processing Systems 10 (1997), pp.

doi:10.1016/0042-6989(93)90156-Q Medline 838844.
18. C. D. Gilbert, T. N. Wiesel, Columnar specificity of intrinsic horizontal and 43. J. Tsotsos, A. Rothenstein, Computational models of visual attention.
corticocortical connections in cat visual cortex. J. Neurosci. 9, 24322442 (1989). Scholarpedia 6, 6201 (2011). doi:10.4249/scholarpedia.6201
Medline 44. B. A. Olshausen, C. H. Anderson, D. C. Van Essen, A neurobiological model of visual
19. E. Craft, H. Schtze, E. Niebur, R. von der Heydt, A neural model of figure-ground attention and invariant pattern recognition based on dynamic routing of
organization. J. Neurophysiol. 97, 43104326 (2007). information. J. Neurosci. 13, 47004719 (1993). Medline
doi:10.1152/jn.00203.2007 Medline 45. Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to
20. V. A. F. Lamme, V. Rodriguez-Rodriguez, H. Spekreijse, Separate processing document recognition. Proc. IEEE 86, 22782324 (1998). doi:10.1109/5.726791
dynamics for texture elements, boundaries and surfaces in primary visual cortex 46. K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale
of the macaque monkey. Cereb. Cortex 9, 406413 (1999). image recognition, paper presented at the International Conference on Learning
doi:10.1093/cercor/9.4.406 Medline Representations (ICLR) 2015, San Diego, CA, 7 to 9 May 2015.
21. E. A. DeYoe, D. C. Van Essen, Concurrent processing streams in monkey visual 47. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, F.-F. Li, Imagenet: A large-scale
cortex. Trends Neurosci. 11, 219226 (1988). doi:10.1016/0166-2236(88)90130- hierarchical image database, in IEEE Conference on Computer Vision and Pattern
0 Medline Recognition (IEEE, 2009), pp. 248255.
22. X. Huang, M. A. Paradiso, V1 response timing and surface filling-in. J. Neurophysiol. 48. A. Wong, A. L. Yuille, One shot learning via compositions of meaningful patches,
100, 539547 (2008). doi:10.1152/jn.00997.2007 Medline in 2015 IEEE International Conference on Computer Vision (IEEE, 2015), pp. 1197
23. H. Zhou, H. S. Friedman, R. von der Heydt, Coding of border ownership in monkey 1205.
visual cortex. J. Neurosci. 20, 65946611 (2000). Medline 49. D. P. Kingma, M. Welling, Stochastic gradient VB and the variational auto-
24. S.-C. Zhu, D. Mumford, A Stochastic Grammar of Images (Now Publishers, 2007). encoder, paper presented at the International Conference on Learning
25. L. L. Zhu, Y. Chen, A. Yuille, Recursive compositional models for vision: Description Representations (ICLR) 2014, Banff, Canada, 14 to 16 April 2014.
and review of recent work. J. Math. Imaging Vis. 41, 122146 (2011). 50. K. Gregor, I. Danihelka, A. Graves, D. J. Rezende, D. Wierstra, DRAW: A recurrent
doi:10.1007/s10851-011-0282-2 neural network for image generation, in Proceedings of the 32nd International
26. A. L. Yuille, Towards a theory of compositional learning and encoding of objects, Conference on Machine Learning (Proceedings of Machine Learning Research,
in 2011 IEEE International Conference on Computer Vision Workshops (ICCV 2015), pp. 14621471.
Workshops) (IEEE, 2011), pp. 14481455. 51. C. K. Williams, M. K. Titsias, Greedy learning of multiple objects in images using
27. R. Salakhutdinov, J. B. Tenenbaum, A. Torralba, Learning to learn with compound robust statistics and factorial learning. Neural Comput. 16, 10391062 (2004).
HD models, in Advances in Neural Information Processing Systems 24 (2012), pp. doi:10.1162/089976604773135096 Medline
19. 52. T. Gao, B. Packer, D. Koller, A segmentation-aware object detection model with
28. Y. Chen, L. Zhu, C. Lin, A. Yuille, H. Zhang, Rapid inference on a novel AND/OR occlusion handling, in 2011 IEEE Computer Society Conference on Computer
graph for object detection, segmentation and parsing, in Advances in Neural Vision and Pattern Recognition (IEEE, 2011), pp. 13611368.
Information Processing Systems 20 (2007), pp. 289296. 53. R. Fransens, C. Strecha, L. Van Gool, A mean field EM-algorithm for coherent
29. L. Zhu, A. Yuille, Recursive compositional models: Representation, learning, and occlusion handling in MAP-estimation problems, in 2006 IEEE Computer Society
inference, In 2009 IEEE Computer Society Conference on Computer Vision and Conference on Computer Vision and Pattern Recognition, vol. 1 (IEEE, 2006) pp.
Pattern Recognition (IEEE, 2009), p. 5. 300307.
30. Z. Tu, X. Chen, A. L. Yuille, S.-C. Zhu, Image parsing: Unifying segmentation, 54. D. Karatzas, F. Shafait, S. Uchida, M. Iwamura, L. Gomez i Bigorda, S. R. Mestre, J.
detection, and recognition. Int. J. Comput. Vis. 63, 113140 (2005). Mas, D. F. Mota, J. A. Almazan, L.-P. de las Heras, ICDAR 2013 Robust Reading
doi:10.1007/s11263-005-6642-x Competition, in 2013 12th International Conference on Document Analysis and
31. S. Fidler, A. Leonardis, Towards scalable representations of object categories: Recognition (IEEE, 2013), pp. 14841493.
Learning a hierarchy of parts, in 2007 IEEE Computer Society Conference on 55. M. Jaderberg, K. Simonyan, A. Vedaldi, A. Zisserman, Deep structured output
Computer Vision and Pattern Recognition (IEEE, 2007). learning for unconstrained text recognition, paper presented at the International
32. Y. Jin, S. Geman, Context and hierarchy in a probabilistic image model, in 2006 Conference on Learning Representations (ICLR) 2015, San Diego, CA, 7 to 9 May
IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2015.
vol. 2 (IEEE, 2006), pp. 21452152. 56. J. Wang, A. Yuille, Semantic Part segmentation using compositional model
33. Supplementary materials. combining shape and appearance, in 2015 IEEE Conference on Computer Vision
34. Z. Si, S.-C. Zhu, Learning AND-OR templates for object recognition and detection. and Pattern Recognition (IEEE, 2015), pp. 17881797.
IEEE Trans. Pattern Anal. Mach. Intell. 35, 21892205 (2013). 57. D. Tabernik, A. Leonardis, M. Boben, D. Skoaj, M. Kristan, Adding discriminative
35. S. Geman, Invariance and selectivity in the ventral visual pathway. J. Physiol. Paris power to a generative hierarchical compositional model using histograms of
compositions. Comput. Vis. Image Underst. 138, 102113 (2015). Trends Cogn. Sci. 11, 5864 (2007). doi:10.1016/j.tics.2006.11.009 Medline
doi:10.1016/j.cviu.2015.04.006 83. L. Bottou, Y. Bengio, Y. Le Cun, Global training of document processing systems
58. S. M. Ali Eslami, N. Heess, T. Weber, Y. Tassa, K. Kavukcuoglu, Attend, infer, using graph transformer networks, in 1997 IEEE Conference on Computer Vision
repeat: Fast scene understanding with generative models, paper presented at and Pattern Recognition (IEEE, 1997), pp. 489494.
the 30th Conference on Neural Information Processing Systems (NIPS 2016), 84. Z. Kourtzi, N. Kanwisher, Representation of perceived object shape by the human
Barcelona, Spain, 5 to 10 December 2016. lateral occipital complex. Science 293, 15061509 (2001).
59. M. Lazaro-Gredilla, Y. Liu, D. S. Phoenix, D. George, Hierarchical compositional doi:10.1126/science.1061133 Medline
feature learning, arXiv:1611.02252 [cs.LG] (7 November 2016). 85. J. J. DiCarlo, D. Zoccolan, N. C. Rust, How does the brain solve visual object
60. S. Ullman, Visual routines. Cognition 18, 97159 (1984). doi:10.1016/0010- recognition? Neuron 73, 415434 (2012). doi:10.1016/j.neuron.2012.01.010
0277(84)90023-4 Medline Medline
61. S. Litvak, S. Ullman, Cortical circuitry implementing graphical models. Neural 86. D. H. Hubel, T. N. Wiesel, Receptive fields, binocular interaction and functional
Comput. 21, 30103056 (2009). doi:10.1162/neco.2009.05-08-783 Medline architecture in the cats visual cortex. J. Physiol. 160, 106154 (1962).
62. D. George, J. Hawkins, Towards a mathematical theory of cortical micro-circuits. doi:10.1113/jphysiol.1962.sp006837 Medline
PLOS Comput. Biol. 5, e1000532 (2009). doi:10.1371/journal.pcbi.1000532 87. L. Wiskott, T. J. Sejnowski, Slow feature analysis: Unsupervised learning of
Medline invariances. Neural Comput. 14, 715770 (2002).
63. N. Le Roux, N. Heess, J. Shotton, J. Winn, Learning a generative model of images doi:10.1162/089976602317318938 Medline
by factoring appearance and shape. Neural Comput. 23, 593650 (2011). 88. R. D. S. Raizada, S. Grossberg, Towards a theory of the laminar architecture of
doi:10.1162/NECO_a_00086 Medline cerebral cortex: Computational clues from the visual system. Cereb. Cortex 13,

64. S. Eslami, C. Williams, A generative model for parts-based object segmentation, 100113 (2003). doi:10.1093/cercor/13.1.100 Medline
in Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. 89. O. Ben-Shahar, S. Zucker, Geometrical computations explain projection patterns
Burges, L. Bottou, K. Q. Weinberger, Eds. (Curran Associates, 2012), pp. 100107. of long-range horizontal connections in visual cortex. Neural Comput. 16, 445
65. A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep 476 (2004). doi:10.1162/089976604772744866 Medline
convolutional neural networks, in Advances in Neural Information Processing 90. J. Hawkins, D. George, J. Niemasik, Sequence memory for prediction, inference
Systems 25 (2012), pp. 10971105. and behaviour. Philos. Trans. R. Soc. Lond. B Biol. Sci. 364, 12031209 (2009).
66. A. Globerson, T. S. Jaakkola, Fixing max-product: Convergent message passing doi:10.1098/rstb.2008.0322 Medline
algorithms for MAP LP-relaxations, in Advances in Neural Information Processing 91. K. Moutoussis, The physiology and psychophysics of the color-form relationship:
Systems 20 (2008), pp. 553560. A review. Front. Psychol. 6, 1407 (2015). Medline
67. V. Kolmogorov, Convergent tree-reweighted message passing for energy 92. E. A. Lachica, P. D. Beck, V. A. Casagrande, Parallel pathways in macaque monkey
minimization. IEEE Trans. Pattern Anal. Mach. Intell. 28, 15681583 (2006). striate cortex: Anatomically defined columns in layer III. Proc. Natl. Acad. Sci.
68. H. Wang, K. Daphne, Subproblem-tree calibration: A unified approach to max- U.S.A. 89, 35663570 (1992). doi:10.1073/pnas.89.8.3566 Medline
product message passing. In Proceedings of the 30th International Conference on 93. F. Federer, J. M. Ichida, J. Jeffs, I. Schiessl, N. McLoughlin, A. Angelucci, Four
Machine Learning (Proceedings of Machine Learning Research, 2013), pp. 190 projection streams from primate V1 to the cytochrome oxidase stripes of V2. J.
198). Neurosci. 29, 1545515471 (2009). doi:10.1523/JNEUROSCI.1648-09.2009
69. S. E. Shimony, Finding MAPs for belief networks is NP-hard. Artif. Intell. 68, 399 Medline
410 (1994). doi:10.1016/0004-3702(94)90072-8 94. C. W. Tyler, R. von der Heydt, Contour-, surface-, and object-related coding in the
70. T. Werner, A linear programming approach to max-sum problem: A review. IEEE visual cortex, in Computer Vision: From Surfaces to 3D Objects (Chapman and
Trans. Pattern Anal. Mach. Intell. 29, 11651179 (2007). Hall/CRC, 2011), pp. 145162.
doi:10.1109/TPAMI.2007.1036 Medline 95. F. T. Qiu, R. von der Heydt, Neural representation of transparent overlay. Nat.
71. N. Komodakis, N. Paragios, G. Tziritas, MRF energy minimization and beyond via Neurosci. 10, 283284 (2007). doi:10.1038/nn1853 Medline
dual decomposition. IEEE Trans. Pattern Anal. Mach. Intell. 33, 531552 (2011). 96. K. Friston, The free-energy principle: A unified brain theory? Nat. Rev. Neurosci.
72. T. Meltzer, A. Globerson, Y. Weiss, Convergent message passing algorithms a 11, 127138 (2010). doi:10.1038/nrn2787 Medline
unifying view, In Proceedings of the Twenty-Fifth Conference on Uncertainty in 97. D. J. Felleman, D. C. Van Essen, Distributed hierarchical processing in the primate
Artificial Intelligence, J. A. Bilmes, A. Y. Ng, Eds. (AUAI Press, 2009), pp. 393401 cerebral cortex. Cereb. Cortex 1, 147 (1991). doi:10.1093/cercor/1.1.1 Medline
73. S. G. Mallat, Zhifeng Zhang, Matching pursuits with time-frequency dictionaries. 98. A. M. Bastos, W. M. Usrey, R. A. Adams, G. R. Mangun, P. Fries, K. J. Friston, M.
IEEE Trans. Signal Process. 41, 33973415 (1993). doi:10.1109/78.258082 Andre, Canonical microcircuits for predictive coding. Neuron 76, 695711 (2013).
74. S. Fidler, M. Boben, A. Leonardis, Optimization framework for learning a 99. X. Lou, K. Kansky, W. Lehrach, C. C. Laan, B. Marthi, D. S. Phoenix, D. George,
hierarchical shape vocabulary for object class detection, in BMVC 2009 (2009). Generative shape models: Joint text recognition and segmentation with very little
75. B. Li et al., Shrec12 track: Generic 3d shape retrieval, paper presented at the training data, in Advances in Neural Information Processing Systems 29 (2016).
Eurographics Workshop on 3D Object Retrieval, Cagliari, Italy, 13 May 2012. 100. L. von Ahn, B. Maurer, C. McMillen, D. Abraham, M. Blum, reCAPTCHA: Human-
76. A. Yuille, R. Mottaghi, Complexity of representation and inference in compositional based character recognition via Web security measures. Science 321, 14651468
models with part sharing. J. Mach. Learn. Res. 17, 128 (2016). (2008). doi:10.1126/science.1160379 Medline
77. M. A. Fischler, R. A. Elschlager, The representation and matching of pictorial 101. K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in
structures. IEEE Trans. Comput. 22, 6792 (1973). 2016 IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2016),
78. P. F. Felzenszwalb, D. P. Huttenlocher, Pictorial Structures for Object Recognition. pp. 770778.
Int. J. Comput. Vis. 61, 5579 (2005). doi:10.1023/B:VISI.0000042934.15159.49 102. M. Abadi et al., TensorFlow: Large-scale machine learning on heterogeneous
79. L. L. Zhu, Y. Chen, A. Torralba, W. Freeman, A. Yuille, Part and appearance distributed systems. arXiv:1603.04467 [cs.DC] (14 March 2016).
sharing: Recursive compositional models for multi-view multi-object detection, 103. I. Tsochantaridis, T. Hofmann, T. Joachims, Y. Altun, Support vector machine
in 2010 IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2010). learning for interdependent and structured output spaces, in Proceedings of the
80. C. K. I. Williams, N. J. Adams, DTs: Dynamic trees, in Advances in Neural Twenty-First International Conference on Machine Learning (ACM, 2004), p. 104.
Information Processing Systems 12 (1999), pp. 634640. 104. A. Bissacco, M. Cummins, Y. Netzer, H. Neven, PhotoOCR: Reading text in
81. L. L. Zhu, C. Lin, H. Huang, Y. Chen, A. Yuille, Unsupervised structure learning: uncontrolled conditions, in 2013 IEEE International Conference on Computer
Hierarchical recursive composition, suspicious coincidence and competitive Vision, (IEEE, 2013), pp. 785792.
exclusion, in 10th European Conference on Computer Vision (Springer, 2008), pp. 105. Z. Ghahramani, Probabilistic machine learning and artificial intelligence. Nature
759773. 521, 452459 (2015). doi:10.1038/nature14541 Medline
82. S. Ullman, Object recognition and segmentation by a fragment-based hierarchy. 106. N. D. Goodman, J. B. Tenenbaum, T. Gerstenberg, Concepts in a probabilistic
language of thought, in The Conceptual Mind: New Directions in the Study of
Concepts, E. Margolis, S. Lawrence, Eds. (MIT Press, 2015), pp. 623654.
107. H. Lee, R. Grosse, R. Ranganath, A. Y. Ng, Convolutional deep belief networks for
scalable unsupervised learning of hierarchical representations, in Proceedings of
the 26th Annual International Conference on Machine Learning (ACM, 2009), pp.
609616.
108. D. Kingma, J. Ba, Adam: A method for stochastic optimization, paper presented
at the International Conference on Learning Representations (ICLR) 2015, San
Diego, CA, 7 to 9 May 2015.
ACKNOWLEDGMENTS
We thank the anonymous reviewers for their careful review and helpful suggestions
that greatly improved the manuscript. We thank B. Olshausen, T. Dean, B. Lake,
and B. Jaros for suggesting improvements after reading early versions of this
manuscript. We are grateful to A. Yuille and F.-F. Li for insightful discussions
leading to this work. Data sets used in the paper are available for download at
www.vicarious.com. The inventions described in this paper are protected by U.S.
patents 9262698, 9373085, 9607262 and 9607263. As text-based CAPTCHAs

are still widely used to protect websites, the scientific benefit of releasing the
source code must be balanced with the potential for it to be used for
circumventing protections that prevent automated interactions with websites.
As a compromise, a simplified reference implementation of RCN algorithms for
the MNIST data set is available at www.vicarious.com/science_rcn.
SUPPLEMENTARY MATERIALS
www.sciencemag.org/cgi/content/full/science.aag2612/DC1
Materials and Methods
Supplementary Text
Figs. S1 to S27
Tables S1 to S15
References (63108)
2 June 2016; accepted 8 September 2017

Published online 26 October 2017
10.1126/science.aag2612
Fig. 1. Flexibility of letterform perception in humans. (A) Humans are good at parsing unfamiliar CAPTCHAs.
(B) The same character shape can be rendered in a wide variety of appearances, and people can detect the A
in these images regardless. (C) Common sense and context affect letterform perception: (i) m vs u and n. (ii)
the same line segments are interpreted as N or S depending on occluder positions. (iii) perception of the shapes
aids the recognition of b,i,s,o,n and b,i,k,e. [Bison logo with permission from Seamus Leonard,
http://www.steadynow.com]
Fig. 2. Structure of the RCN. (A) A
hierarchy generates the contours
of an object, and a Conditional
Random Field (CRF) generates its
surface appearance. (B) Two
subnetworks at the same level of
the contour hierarchy keep
separate lateral connections by
making parent-specific copies of
child features and connecting them
with parent-specific laterals; nodes
within the green rectangle are
copies of the feature marked e.
(C) A three level RCN representing
the contours of a square. Features
at Level 2 represent the four

corners, and each corner is
represented as a conjunction of
four line-segment features. (D)
Four-level network representing an
A.
Fig. 3. Samples from RCN. (A)
Samples from a corner feature with and
without lateral connections. (B)
Samples from character A for
different deformability settings,
determined by pooling and lateral
perturb-factors, in a 3-level hierarchy
similar to Fig. 2D, where the lowest level
features are edges. Column 2 shows a
balanced setting where deformability is
distributed between the levels to
produce local deformations and global
translations. The other columns show
some extreme configurations. (C)
Contour to surface-CRF interaction for
a cube. Green factors: foreground-to-
background edges, blue: within-object

edges. (D) Different surface-
appearance samples for the cubical
shape in C. [See section 3 of (33) for
CRF parameters.]
Fig. 4. (A) (i) Forward pass, including lateral propagation, produces
hypotheses about the multiple letters present in the input image. PreProc is
a bank of Gabor-like filters that convert from pixels to edge likelihoods
[section 4.2 of (33)]. (ii) Backward pass and lateral propagation creates the
segmentation mask for a selected forward-pass hypothesis, here the letter
A [section 4.4 of (33)]. (iii) A false hypothesis V is hallucinated at the
intersection of A and K; false hypotheses are resolved via parsing
[section 4.7 of (33)]. (iv) Multiple hypotheses can be activated to produce a
joint explanation that involves explaining away and occlusion reasoning. (B)
Learning features at the second feature level. Colored circles represent
feature activations. The dotted circle is a proposed feature [see text and
section 5 of (33)]. (C) Learning of laterals from contour adjacency (see
text).
Fig. 5. Parsing CAPTCHAs
with RCN. (A) Representative
reCAPTCHA parses showing
top two solutions, their
segmentations, and labels by
two different Amazon
Mechanical Turk workers. (B)
Word accuracy rates of RCN
and CNN on the control
CAPTCHA data set. CNN is
brittle and RCN is robust
when character-spacing is
changed. (C) Accuracies for
different CAPTCHA styles.
(D) Representative
BotDetect parses and
segmentations (indicated by

the different colors).
Fig. 6. MNIST classification results for
training with few examples. (A) MNIST
classification accuracy for RCN, CNN,
and CPM. (B) Classification accuracy on
corrupted MNIST tests. Legends show
the total number of training examples.
(C) MNIST classification accuracy for
different RCN configurations.
Fig. 7. Generation, occlusion reasoning, and scene-text parsing with
RCN. Examples of reconstructions (A) and reconstruction error (B) from
RCN, VAE and DRAW on corrupted MNIST. Legends show the number of
training examples. (C) Occlusion reasoning. The third column shows edges
remaining after RCN explains away the edges of the first detected object.
Ground-truth masks reflect the occlusion relationships between the square
and the digit. The portions of the digit that are in front of the square are
indicated by brown color and the portions that are behind the square are
indicated by orange color. The last column shows the predicted occlusion
mask. (D) One-shot generation from Omniglot. In each column, row 1 shows
the training example and the remaining rows show generated samples. (E)
Examples of ICDAR images successfully parsed by RCN. The yellow outlines
show segmentations.
Fig. 8. Application of RCN to
parsing scenes with objects.
Shown are the detections and
instance segmentations obtained
when RCN was applied to a scene
parsing task with multiple real-
world objects in cluttered scenes
on random backgrounds. Our
experiments suggest that RCN
could be generalized beyond text
parsing [see section 8.12 of (33)
and Discussion].
Table 1. Accuracy and number of training images for different methods on the ICDAR-13 robust reading data set.
Method Accuracy Total no. of training images

PLT (54) 64.6% Unknown
NSEP (54) 63.7% Unknown
PicRead (54) 63.1% Unknown
Deep Structured Output Learning (55) 81.8% 8,000,000
PhotoOCR (54) 84.3% 7,900,000
RCN 86.2% 26,214 (reduced to 1406)
A generative vision model that trains with high data efficiency and breaks text-based CAPTCHAs
D. George, W. Lehrach, K. Kansky, M. Lzaro-Gredilla, C. Laan, B. Marthi, X. Lou, Z. Meng, Y. Liu, H. Wang, A. Lavin and D. S.
Phoenix
published online October 26, 2017

ARTICLE TOOLS http://science.sciencemag.org/content/early/2017/10/25/science.aag2612
SUPPLEMENTARY http://science.sciencemag.org/content/suppl/2017/10/25/science.aag2612.DC1
MATERIALS
REFERENCES This article cites 57 articles, 12 of which you can access for free
http://science.sciencemag.org/content/early/2017/10/25/science.aag2612#BIBL
PERMISSIONS http://www.sciencemag.org/help/reprints-and-permissions
Use of this article is subject to the Terms of Service
Science (print ISSN 0036-8075; online ISSN 1095-9203) is published by the American Association for the Advancement of
Science, 1200 New York Avenue NW, Washington, DC 20005. 2017 The Authors, some rights reserved; exclusive licensee
American Association for the Advancement of Science. No claim to original U.S. Government Works. The title Science is a
registered trademark of AAAS.

A Generative Vision Model That Trains With High Data Efficiency and Breaks Text-Based Captchas

Uploaded by

Copyright:

Available Formats

A Generative Vision Model That Trains With High Data Efficiency and Breaks Text-Based Captchas

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

A Generative Vision Model That Trains With High Data Efficiency and Breaks Text-Based Captchas

Uploaded by

Copyright:

Available Formats

RESEARCH ARTICLES

Cite as: D. George et al., Science

A generative vision model that trains with high data

Downloaded from http://science.sciencemag.org/ on October 26, 2017

Downloaded from http://science.sciencemag.org/ on October 26, 2017

Downloaded from http://science.sciencemag.org/ on October 26, 2017

Downloaded from http://science.sciencemag.org/ on October 26, 2017

Downloaded from http://science.sciencemag.org/ on October 26, 2017

Downloaded from http://science.sciencemag.org/ on October 26, 2017

Downloaded from http://science.sciencemag.org/ on October 26, 2017

Downloaded from http://science.sciencemag.org/ on October 26, 2017

Downloaded from http://science.sciencemag.org/ on October 26, 2017

2 June 2016; accepted 8 September 2017

Downloaded from http://science.sciencemag.org/ on October 26, 2017

Downloaded from http://science.sciencemag.org/ on October 26, 2017

Downloaded from http://science.sciencemag.org/ on October 26, 2017

Downloaded from http://science.sciencemag.org/ on October 26, 2017

Downloaded from http://science.sciencemag.org/ on October 26, 2017

Method Accuracy Total no. of training images

Downloaded from http://science.sciencemag.org/ on October 26, 2017

published online October 26, 2017

Downloaded from http://science.sciencemag.org/ on October 26, 2017

Use of this article is subject to the Terms of Service

You might also like