Practical Block-wise Neural Network Architecture Generation

Zhao Zhong1,3∗, Junjie Yan2 ,Wei Wu2 ,Jing Shao2 ,Cheng-Lin Liu1,3,4
1
National Laboratory of Pattern Recognition,Institute of Automation, Chinese Academy of Sciences
2
SenseTime Research 3 University of Chinese Academy of Sciences
4
CAS Center for Excellence of Brain Science and Intelligence Technology
Email: {zhao.zhong, liucl}@nlpr.ia.ac.cn, {yanjunjie, wuwei, shaojing}@sensetime.com
arXiv:1708.05552v3 [cs.CV] 14 May 2018

Abstract performance network architecture generally possesses a

tremendous number of possible configurations about the
Convolutional neural networks have gained a remark- number of layers, hyperparameters in each layer and type
able success in computer vision. However, most usable net- of each layer. It is hence infeasible for manually exhaus-
work architectures are hand-crafted and usually require ex- tive search, and the design of successful hand-crafted net-
pertise and elaborate design. In this paper, we provide a works heavily rely on expert knowledge and experience.
block-wise network generation pipeline called BlockQNN Therefore, constructing the network in a smart and auto-
which automatically builds high-performance networks us- matic manner remains an open problem.
ing the Q-Learning paradigm with epsilon-greedy explo- Although some recent works have attempted computer-
ration strategy. The optimal network block is constructed by aided or automated network design [2, 37], there are sev-
the learning agent which is trained sequentially to choose eral challenges still unsolved: (1) Modern neural networks
component layers. We stack the block to construct the whole always consist of hundreds of convolutional layers, each
auto-generated network. To accelerate the generation pro- of which has numerous options in type and hyperparame-
cess, we also propose a distributed asynchronous frame- ters. It makes a huge search space and heavy computational
work and an early stop strategy. The block-wise genera- costs for network generation. (2) One typically designed
tion brings unique advantages: (1) it performs competitive network is usually limited on a specific dataset or task, and
results in comparison to the hand-crafted state-of-the-art thus is hard to transfer to other tasks or generalize to another
networks on image classification, additionally, the best net- dataset with different input data sizes. In this paper, we pro-
work generated by BlockQNN achieves 3.54% top-1 error vide a solution to the aforementioned challenges by a novel
rate on CIFAR-10 which beats all existing auto-generate fast Q-learning framework, called BlockQNN, to automati-
networks. (2) in the meanwhile, it offers tremendous re- cally design the network architecture, as shown in Fig. 1.
duction of the search space in designing networks which Particularly, to make the network generation efficient
only spends 3 days with 32 GPUs, and (3) moreover, it has and generalizable, we introduce the block-wise network
strong generalizability that the network built on CIFAR also generation, i.e., we construct the network architecture as a
performs well on a larger-scale ImageNet dataset. flexible stack of personalized blocks rather tedious per-layer
network piling. Indeed, most modern CNN architectures
such as Inception [30, 14, 31] and ResNet Series [10, 11]
1. Introduction are assembled as the stack of basic block structures. For
example, the inception and residual blocks shown in Fig. 1
During the last decades, Convolutional Neural Networks are repeatedly concatenated to construct the entire network.
(CNNs) have shown remarkable potentials almost in ev- With such kind of block-wise network architecture, the
ery field in the computer vision society [17]. For exam- generated network owns a powerful generalization to other
ple, thanks to the network evolution from AlexNet [16], task domains or different datasets.
VGG [25], Inception [30] to ResNet [10], the top-5 per-
In comparison to previous methods like NAS [37] and
formance on ImageNet challenge steadily increases from
MetaQNN [2], as depicted in Fig. 1, we present a more
83.6% to 96.43%. However, as the performance gain
readily and elegant model generation method that specif-
usually requires an increasing network capacity, a high-
ically designed for block-wise generation. Motivated by
∗ This work was done when Zhao Zhong worked as an intern at Sense- the unsupervised reinforcement learning paradigm, we em-
Time Research. ploy the well-known Q-learning [33] with experience re-
 Hand-crafted Network Auto-generated Network
AlexNet VGGNet Inception-block Residue-block NAS MetaQNN BlockQNN

Input Input
Input Input Input
Input
3x3 Conv
11x11 Conv 3x3 Conv 5x5 Conv
Conv 3x3 Conv
Pool 1/2 3x3 Conv
Conv,1 Conv,1 MaxP,3 BN 5x5 Conv
5x5 Conv Dropout 1/8 Conv,1 Conv,3 Conv,5 Conv,3
Conv,1 3x7 Conv
ReLU
Pool 1/2 7x7 Conv 2x2 MaxP
Conv,5 AvgP,3
Conv,3 Conv,5 Conv,1 Conv 7x7 Conv
3x3 Conv 1x1 Conv
7x3 Conv
BN Add Concat
3x3 Conv 7x1 Conv Dropout 1/4

3x3 Conv 7x7 Conv 5x5 Conv Conv,3

Add
5x7 Conv
Concat
Pool 1/2 ReLU 3x3 MaxP
7x7 Conv

Linear 4096 7x5 Conv Dropout 3/8

Output Output Concat
7x5 Conv
Linear 4096 3x3 Conv
7x5 Conv Output
Linear Linear Linear

Figure 1. The proposed BlockQNN (right in red box) compared with the hand-crafted networks marked in yellow and the existing auto-
generated networks in green. Automatically generating the plain networks [2, 37] marked in blue need large computational costs on
searching optimal layer types and hyperparameters for each single layer, while the block-wise network heavily reduces the cost to search
structures only for one block. The entire network is then constructed by stacking the generated blocks. Similar block concept has been
demonstrated its superiority in hand-crafted networks, such as inception-block and residue-block marked in red.

play [19] and epsilon-greedy strategy [21] to effectively and be transferred to ImageNet with little modification but
efficiently search the optimal block structure. The network still achieve outstanding performance.
block is constructed by the learning agent which is trained
sequentiality to choose component layers. Afterwards we 2. Related Work
stack the block to construct the whole auto-generated net-
Early works, from 1980s, have made efforts on automat-
work. Moreover, we propose an early stop strategy to en-
ing neural network design which often searched good archi-
able efficient search with fast convergence. A novel reward
tecture by the genetic algorithm or other evolutionary algo-
function is designed to ensure the accuracy of the early
rithms [24, 27, 26, 28, 23, 7, 34]. Nevertheless, these works,
stopped network having positive correlation with the con-
to our best knowledge, cannot perform competitively com-
verged network. We can pick up good blocks in reduced
pared with hand-crafted networks. Recent works, i.e. Neu-
training time using this property. With this acceleration
ral Architecture Search (NAS) [37] and MetaQNN [2],
strategy, we can construct a Q-learning agent to learn the
adopted reinforcement learning to automatically search a
optimal block-wise network structures for a given task with
good network architecture. Although they can yield good
limited resources (e.g. few GPUs or short time period). The
performance on small datasets such as CIFAR-10, CIFAR-
generated architectures are thus succinct and have powerful
100, the direct use of MetaQNN or NAS for architecture
generalization ability compared to the networks generated
design on big datasets like ImageNet [6] is computationally
by the other automatic network generation methods.
expensive via searching in a huge space. Besides, the net-
The proposed block-wise network generation brings a
work generated by this kind of methods is task-specific or
few advantages as follows:
dataset-specific, that is, it cannot been well transferred to
• Effective. The automatically generated networks other tasks nor datasets with different input data sizes. For
present comparable performances to those of hand- example, the network designed for CIFAR-10 cannot been
crafted networks with human expertise. The proposed generalized to ImageNet.
method is also superior to the existing works and Instead, our approach is aimed to design network block
achieves a state-of-the-art performance on CIFAR-10 architecture by an efficient search method with a dis-
with 3.54% error rate. tributed asynchronous Q-learning framework as well as an
• Efficient. We are the first to consider block-wise early-stop strategy. The block design conception follows
setup in automatic network generation. Companied the modern convolutional neural networks such as Incep-
with the proposed early stop strategy, the proposed tion [30, 14, 31] and Resnet [10, 11]. The inception-based
method results in a fast search process. The network networks construct the inception blocks via a hand-
generation for CIFAR task reaches convergence with crafted multi-level feature extractor strategy by computing
only 32 GPUs in 3 days, which is much more efficient 1 × 1, 3 × 3, and 5 × 5 convolutions, while the Resnet uses
than that by NAS [37] with 800 GPUs in 28 days. residue blocks with shortcut connection to make it
• Transferable. It offers surprisingly superior transfer- easier to represent the identity mapping which allows a very
able ability that the network generated for CIFAR can deep network. The blocks automatically generated by our
 Name Index Type Kernel Size Pred1 Pred2
Input Input
Convolution T 1 1, 3, 5 K 0
Max Pooling T 2 1, 3 K 0 Conv Conv
Average Pooling T 3 1, 3 K 0 Pooling
Block ×N
Identity T 4 0 K 0
Conv
Elemental Add T 5 0 K K Pooling
Pooling
Concat T 6 0 K K Block ×N
Block ×N
Terminal T 7 0 0 0
Pooling Pooling
Table 1. Network Structure Code Space. The space contains seven
Block ×N Block ×N
types of commonly used layers. Layer index stands for the posi-
tion of the current layer in a block, the range of the parameters is Pooling
Linear
set to be T = {1, 2, 3, ...max layer index}. Three kinds of ker- Block ×N
nel sizes are considered for convolution layer and two sizes for
Pooling
pooling layer. Pred1 and Pred2 refer to the predecessor parame-
ters which is used to represent the index of layers predecessor, the Block ×N
allowed range is K = {1, 2, ..., current layer index − 1} Linear

Input Input Figure 3. Auto-generated networks on CIFAR-10 (left) and Im-

Identity Identity ageNet (right). Each network starts with a few convolution lay-
ers to learn low-level features, and followed by multiple repeated
blocks with several pooling layers inserted to downsample.
Conv,1 Conv,1 MaxP,3 Conv,3

Conv,3 Conv,5 Conv,1 Conv,3 similar structure but with different weights and filter num-
bers to construct the network. With the block-wise design,
Concat Add the network can not only achieves high performance but
also has powerful generalization ability to different datasets
Concat Output
and tasks. Unlike previous research on automating neural
Output network design which generate the entire network directly,
Codes = [(1,4,0,0,0), (2,1,1,1,0), (3,1,3,2,0), Codes = [(1,4,0,0,0), (2,1,3,1,0),
we aim at designing the block structure.
(4,1,1,1,0), (5,1,5,4,0), (6,6,0,3,5), (3,1,3,2,0), (4,5,0,1,3),
(7,2,3,1,0), (8,1,1,7,0), (9,6,0,6,8), (5,7,0,0,0)] As a CNN contains a feed-forward computation proce-
(10,7,0,0,0)] dure, we represent it by a directed acyclic graph (DAG),
Figure 2. Representative block exemplars with their Network where each node corresponds to a layer in the CNN while
structure codes (NSC) respectively: the block with multi-branch directed edges stand for data flow from one layer to another.
connections (left) and the block with shortcut connections (right). To turn such a graph into a uniform representation, we pro-
pose a novel layer representation called Network Structure
approach have similar structures such as some blocks con- Code (NSC), as shown in Table 2. Each block is then de-
tain short cut connections and inception-like multi-branch picted by a set of 5-D NSC vectors. In NSC, the first three
combination. We will discuss the details in Section 5.1. numbers stand for the layer index, operation type and kernel
Another bunch of related works include hyper-parameter size. The last two are predecessor parameters which refer
optimization [3], meta-learning [32] and learning to learn to the position of a layer’s predecessor layer in structure
methods [12, 1]. However, the goal of these works is to codes. The second predecessor (Pred2) is set for the layer
use meta-data to improve the performance of the existing owns two predecessors, and for the layer with only one pre-
algorithms, such as finding the optimal learning rate of op- decessor, Pred2 will be set to zero. This design is motivated
timization methods or the optimal number of hidden layers by the current powerful hand-crafted networks like Incep-
to construct the network. In this paper, we focus on learn- tion and Resnet which own their special block structures.
ing the entire topological architecture of network blocks to This kind of block structure shares similar properties such
improve the performance. as containing more complex connections, e.g. shortcut con-
nections or multi-branch connections, than the simple con-
3. Methodology nections of the plain network like AlexNet. Thus, the pro-
posed NSC can encode complexity architectures as shown
3.1. Convolutional Neural Network Blocks
in Fig. 2. In addition, all of the layer without successor in
The modern CNNs, e.g. Inception and Resnet, are de- the block are concatenated together to provide the final out-
signed by stacking several blocks each of which shares put. Note that each convolution operation, same as the dec-
 (1,1,1,0,0) (2,1,3,1,0) (3,1,3,1,0) … (T,x,x,x,x) Input
Agent samples
structure codes

Stack blocks

…
Update
Input Conv,1x1 to generate a
Q-Value network

(1,x,x,x,x) (2,x,x,x,x) (3,x,x,x,x) … (T,x,x,x,x) Conv,3x3 Feedback

validation Train the
State
Action
(1,7,0,0,0) (2,7,0,0,0) (3,7,0,0,0) … (T,7,0,0,0) Block
accuracy as
reward
network on a
task

(a) (b) (c)

Figure 4. Q-learning process illustration. (a) The state transition process by different action choices. The block structure in (b) is generated
by the red solid line in (a). (c) The flow chart of the Q-learning procedure.

laration in Resnet [11], refers to a Pre-activation Convolu- to the defined NSC set with a limited number of choices,
tional Cell (PCC) with three components, i.e. ReLU, Con- both the state and action space are thus finite and discrete
volution and Batch Normalization. This results in a smaller to ensure a relatively small searching space. The state tran-
searching space than that with three components separate sition process (st , a(st )) → (st+1 ) is shown in Fig. 4(a),
search, and hence with the PCC, we can get better initial- where t refers to the current layer. The block example in
ization for searching and generating optimal block structure Fig. 4(b) is generated by the red solid lines in Fig. 4(a). The
with a quick training process. learning agent is given the task of sequentially picking NSC
Based on the above defined blocks, we construct of a block. The structure of block can be considered as an
the complete network by stacking these block structures action selection trajectory τa1:T , i.e. a sequence of NSCs.
sequentially which turn a common plain network into We model the layer selection process as a Markov Decision
its counterpart block version. Two representative auto- Process with the assumption that a well-performing layer in
generated networks on CIFAR and ImageNet tasks are one block should also perform well in another block [2]. To
shown in Fig. 3. There is no down-sampling operation find the optimal architecture, we ask our agent to maximize
within each block. We perform down-sampling directly its expected reward over all possible trajectories, denoted
by the pooling layer. If the size of feature map is halved by Rτ ,
by pooling operation, the block’s weights will be doubled.
The architecture for ImageNet contains more pooling layers Rτ = EP (τa1:T ) [R], (1)
than that for CIFAR because of their different input sizes,
where the R is the cumulative reward. For this maximiza-
i.e. 224 × 224 for ImageNet and 32 × 32 for CIFAR. More
tion problem, it is usually to use recursive Bellman Equation
importantly, the blocks can be repeated any N times to
to optimality. Given a state st ∈ S, and subsequent action
fulfill different demands, and even place the blocks in other
a ∈ A(st ), we define the maximum total expected reward
manner, such as inserting the block into the Network-in-
to be Q∗ (st , a) which is known as Q-value of state-action
Network [20] framework or setting short cut connection be-
pair. The recursive Bellman Equation then can be written as
tween different blocks.
Q∗ (st , a) = Est+1 |st ,a [Er|st ,a,st+1 [r|st , a, st+1 ]
3.2. Designing Network Blocks With Q-Learning
+γ max Q∗ (st+1 , a0 )]. (2)
Albeit we squeeze the search space of the entire network a0 ∈A(st+1 ))
design by focusing on constructing network blocks, there
An empirical assumption to solve the above quantity is
is still a large amount of possible structures to seek. There-
to formulate it as an iterative update:
fore, we employ reinforcement learning rather than random
sampling for automatic design. Our method is based on Q- Q(sT , a) =0 (3)
learning, a kind of reinforcement learning, which concerns
Q(sT −1 , aT ) =(1 − α)Q(sT −1 , aT ) + αrT (4)
how an agent ought to take actions so as to maximize the
cumulative reward. The Q-learning model consists of an Q(st , a) =(1 − α)Q(st , a)
agent, states and a set of actions. +α[rt + γ max
0
Q(st+1 , a0 )], t ∈ {1, 2, ...T − 2}, (5)
a
In this paper, the state s ∈ S represents the status of
the current layer which is defined as a Network Structure where α is the learning rate which determines how the
Code (NSC) claimed in Section 3.1, i.e. 5-D vector {layer newly acquired information overrides the old information,
index, layer type, kernel size, pred1, pred2}. The action γ is the discount factor which measures the importance of
a ∈ A is the decision for the next successive layer. Thanks future rewards. rt denotes the intermediate reward observed
 Q-learning Performance with Different Data Analysis of Early Stop Accuracy
intermediate reward Early Stop ACC Final ACC Redefined Reward FLOPs Density
100 5
70
Ignore 𝑟#
Accuracy (%)

shaped reward 𝑟# 95 4

Scalar for FLOPs and Density

65

60

Accuracy (%)
90 3

55
1 6 11 16 21 26 85 2
Iteration (batch)

Figure 5. Comparison results of Q-learning with and without the 80 1

shaped intermediate reward rt . By taking our shaped reward, the
learning process convergent faster than that without shaped reward 75 0
1 11 21 31 41 51 61 71 81 91
start from the same exploration.
Model (block)

Figure 6. The performance of early stop training is poorer than the

for the current state st and sT refers to final state, i.e. termi- final accuracy of a complete training. With the help of FLOPs and
nal layers. rT is the validation accuracy of corresponding Density, it squeezes the gap between the redefined reward function
and the final accuracy.
network trained convergence on training set for aT , i.e. ac-
tion to final state. Since the reward rt cannot be explicitly
measured in our task, we use reward shaping [22] to speed early. In the meanwhile, we notice that the FLOPs and den-
up training. The shaped intermediate reward is defined as: sity of the corresponding blocks have a negative correlation
rT with the final accuracy. Thus, we redefine the reward func-
rt = . (6) tion as
T
Previous works [2] ignore these rewards in the iterative reward = ACCEarlyStop − µ log(FLOPs)
process, i.e. set them to zero, which may cause a slow con- −ρ log(Density), (7)
vergence in the beginning. This is known as the temporal
credit assignment problem which makes RL time consum- where FLOPs [8] refer to an estimation of computational
ing [29]. In this case, the Q-value of sT is much higher than complexity of the block, and Density is the edge number
others in early stage of training and thus leads the agent pre- divided by the dot number in DAG of the block. There are
fer to stop searching at the very beginning, i.e. tend to build two hyperparameters, µ and ρ, to balance the weights of
small block with fewer layers. We show a comparison result FLOPs and Density. With the redefined reward function,
in Fig. 5, the learning process of the agent with our shaped the reward is more relevant to the final accuracy.
reward rt convergent much faster than previous method. With this early stop strategy and small searching space of
We summarize the learning procedure in Fig. 4(c). The network blocks, it just costs 3 days to complete the search-
agent first samples a set of structure codes to build the ing process with only 32 GPUs, which is superior to that
block architecture, based on which the entire network is of [37], spends 28 days with 800 GPUs to achieve the same
constructed by stacking these blocks sequentially. We then performance.
train the generated network on a certain task, and the vali-
dation accuracy is regarded as the reward to update the Q- 4. Framework and Training Details
value. Afterwards, the agent picks another set of structure
codes to get a better block structure. 4.1. Distributed Asynchronous Framework
To speed up the learning of agent, we use a distributed
3.3. Early Stop Strategy
asynchronous framework as illustrated in Fig. 7. It consists
Introducing block-wise generation indeed increases of three parts: master node, controller node and compute
the efficiency. However, it is still time consuming to com- nodes. The agent first samples a batch of block structures in
plete the search process. To further accelerate the learn- master node. Afterwards, we store them in a controller node
ing process, we introduce an early stop strategy. As we all which uses the block structures to build the entire networks
know, early stopping training process might result in a poor and allocates these networks to compute nodes. It can be
accuracy. Fig. 6 shows an example, where the early-stop ac- regarded as a simplified parameter-server [5, 18]. Specif-
curacy in yellow line is much lower than the final accuracy ically, the network is trained in parallel on each of com-
in orange line, which means that some good blocks unfor- pute nodes and returns the validation accuracy as reward by
tunately perform worse than bad blocks when stop training controller nodes to update agent. With this framework, we
 ratio of 5 : 1. We train the network without any data aug-
Master Node mentation procedure. The batch size is set to 256. We
use Adam optimizer [15] with β1 = 0.9, β2 = 0.999,
ε = 10−8 . The initial learning rate is set to 0.001 and is
Controller Node reduced with a factor of 0.2 every 5 epochs. All weights are
initialized as in [9]. If the training result after the first epoch
is worse than the random guess, we reduce the learning rate
Compute Nodes by a factor of 0.4 and restart training, with a maximum of 3
times for restart-operations.
Figure 7. The distributed asynchronous framework. It contains After obtaining one optimal block structure, we build
three parts: master node, controller node and compute nodes. the whole network with stacked blocks and train the net-
work until converging to get the validation accuracy as the
 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 criterion to pick the best network. In this phase, we aug-
Iters 95 7 7 7 10 10 10 10 10 12 ment data with randomly cropping the images with size of
32 × 32 and horizontal flipping. All models use the SGD
Table 2. Epsilon Schedules. The number of iteration the agent
trains at each epsilon() state.
optimizer with momentum rate set to 0.9 and weight decay
set to 0.0005. We start with a learning rate of 0.1 and train
the models for 300 epochs, reducing the learning rate in the
can generate network efficiently on multiple machines with 150-th and 225-th epoch. The batch size is set to 128 and
multiple GPUs. all weights are initialized with MSRA initialization [9].
Transferable BlockQNN. We also evaluate the transfer-
4.2. Training Details ability of the best auto-generated block structure searched
on CIFAR-100 to a smaller dataset, CIFAR-10, with only
Epsilon-greedy Strategy. The agent is trained using Q-
10 classes and a larger dataset, ImageNet, containing 1.2M
learning with experience replay [19] and epsilon-greedy
images with 1000 classes. All the experimental settings are
strategy [21]. With epsilon-greedy strategy, the random ac-
the same as those on the CIFAR-100 stated above. The
tion is taken with probability  and the greedy action is cho-
training is conducted with a mini-batch size of 256 where
sen with probability 1 − . We decrease epsilon from 1.0 to
each image has data augmentation of randomly cropping
0.1 following the epsilon schedule as shown in Table 2 such
and flipping, and is optimized with SGD strategy. The ini-
that the agent can transform smoothly from exploration to
tial learning rate, weight decay and momentum are set as
exploitation. We find that the result goes better with a longer
0.1, 0.0001 and 0.9, respectively. We divide the learning
exploration, since the searching scope would become larger
rate by 10 twice, at the 30-th and 60-th epochs. The net-
and the agent can see more block structures in the random
work is trained with a total of 90 epochs. We evaluate the
exploration period.
accuracy on the test images with center crop.
Experience Replay. Following [2], we employ a replay Our framework is implemented under the PyTorch sci-
memory to store the validation accuracy and block descrip- entific computing platform. We use the CUDA backend
tion after each iteration. Within a given interval, i.e. each and cuDNN accelerated library in our implementation for
training iteration, the agent samples 64 blocks with their high-performance GPU acceleration. Our experiments are
corresponding validation accuracies from the memory and carried out on 32 NVIDIA TitanX GPUs and took about 3
updates Q-value 64 times. days to complete searching.
BlockQNN Generation.
In the Q-learning update process, the learning rate α is 5. Results
set to 0.01 and the discount factor γ is 1. We set the hy-
5.1. Block Searching Analysis
perparameters µ and ρ in the redefined reward function as 1
and 8, respectively. The agent samples 64 sets of NSC vec- Fig. 8(a) provides early stop accuracies over 178 batches
tors at a time to compose a mini-batch and the maximum on CIFAR-100, each of which is averaged over 64 auto-
layer index for a block is set to 23. We train the agent with generated block-wise network candidates within in each
178 iterations, i.e. sampling 11, 392 blocks in total. mini-batch. After random exploration, the early stop ac-
During the block searching phase, the compute nodes curacy grows steadily till converges. The mean accuracy
train each generated network for a fixed 12 epochs on within the period of random exploration is 56% while fi-
CIFAR-100 using the early top strategy as described in Sec- nally achieves 65% in the last stage with  = 0.1. We
tion 3.3. CIFAR-100 contains 60, 000 samples with 100 choose top-100 block candidates and train their respective
classes which are divided into training and test set with the networks to verify the best block structure. We show top-
 68 Block-QNN-A Input Input Input
Mean Accuracy
66

64
Accuracy (%)

62 Conv,5 Conv,1 Conv,3 Conv,5 Conv,3 Conv,1 Conv,3 Conv,5 Conv,3 Conv,3 Conv,3 Conv,1 Conv,1
Block-QNN-B
60
Add MaxP,1 MaxP,1 MaxP,3 Conv,5 AvgP,3 Conv,1 Conv,3 Add Conv,3
58
Conv,3 Conv,1 AvgP,3 Add Concat MaxP,3 Conv,5 MaxP,3 Conv,3
56
Add
54 Conv,3
1 21 41 61 81 101 121 141 161
Iteration (batch)
Random Exploration Start Exploitation Concat Concat Concat

(a) Q-learning performance (b) Block-QNN-A (c) Block-QNN-B (d) Block-QNN-S

Figure 8. (a) Q-learning performance on CIFAR-100. The accuracy goes up with the epsilon decrease and the top models are all found in
the final stage, show that our agent can learn to generate better block structures instead of random searching. (b-c) Topology of the Top-2
block structures generated by our approach. We call them Block-QNN-A and Block-QNN-B. (d) Topology of the best block structures
generated with limited parameters, named Block-QNN-S.

Q-learning Performance with Different Structure Codes Method Depth Para C-10 C-100
70 VGG [25] - 7.25 -
PCC{ReLU,Conv,BN}
separate ReLU,BN,Conv
65 ResNet [10] 110 1.7M 6.61 -
60 Wide ResNet [36] 28 36.5M 4.17 20.5
Accuracy (%)

ResNet (pre-activation) [11] 1001 10.2M 4.62 22.71

55
DenseNet (k = 12) [13] 40 1.0M 5.24 24.42
50
DenseNet (k = 12) [13] 100 7.0M 4.10 20.20
45 DenseNet (k = 24) [13] 100 27.2M 3.74 19.25
40 DenseNet-BC (k = 40) [13] 190 25.6M 3.46 17.18
1 21 41 61 81 101 121 141 161
Iteration (batch)
MetaQNN (ensemble) [2] - - 7.32 -
MetaQNN (top model) [2] - 11.2M 6.92 27.14
Figure 9. Q-learning result with different NSC on CIFAR-100. NAS v1 [37] 15 4.2M 5.50 -
The red line refers to searching with PCC, i.e. combination of NAS v2 [37] 20 2.5M 6.01 -
ReLU, Conv and BN. The blue stands for separate searching with
NAS v3 [37] 39 7.1M 4.47 -
ReLU, BN and Conv. The red line is better than blue from the
NAS v3 more filters [37] 39 37.4M 3.65 -
beginning with a big gap.
Block-QNN-A, N=4 25 - 3.60 18.64
Block-QNN-B, N=4 37 - 3.80 18.72
Block-QNN-S, N=2 19 6.1M 4.38 20.65
2 block structures in Fig. 8(b-c), denoted as Block-QNN-
Block-QNN-S more filters 22 39.8M 3.54 18.06
A and Block-QNN-B. As shown in Fig. 8(a), both top-2
blocks are found in the final stage of the Q-learning pro- Table 3. Block-QNN’s results (error rate) compare with state-of-
cess, which proves the effectiveness of the proposed method the-art methods on CIFAR-10 (C-10) and CIFAR-100 (C-100)
in searching optimal block structures rather than randomly dataset.
searching a large amount of models. Furthermore, we ob-
serve that the generated blocks share similar properties with
those state-of-the-art hand-crafted networks. For example, batch normalization (BN). We show the superiority of the
Block-QNN-A and Block-QNN-B contain short-cut con- PCC, searching a combination of three components, in
nections and multi-branch structures which have been man- Fig. 9, compared to the separate search of each component.
ually designed in residual-based and inception-based net- Searching the three components separately is more likely to
works. Compared to other auto-generated methods, the net- generate “bad” blocks and also needs more searching space
works generated by our approach are more elegant and can and time to pursue “good” blocks.
automatically and effectively reveal the beneficial proper-
ties for optimal network structure. 5.2. Results on CIFAR
To squeeze the searching space, as stated in Section 3.1, Due to the small size of images (i.e. 32 × 32) in CIFAR,
we define a Pre-activation Convolutional Cell (PCC) con- we set block stack number as N = 4. We compare our
sists of three components, i.e. ReLU, convolution and generated best architectures with the state-of-the-art hand-
crafted networks or auto-generated networks in Table 3. Method Best Model on CIFAR10 GPUs Time(days)

Comparison with hand-crafted networks - It shows that our MetaQNN [2] 6.92 10 10
Block-QNN networks outperform most hand-crafted net- NAS [37] 3.65 800 28
works. The DenseNet-BC [13] uses additional 1 × 1 con- Our approach 3.54 32 3
volutions in each composite function and compressive tran-
sition layer to reduce parameters and improve performance, Table 4. The required computing resource and time of our ap-
proach compare with other automatic designing network methods.
which is not adopted in our design. Our performance can
be further improved by using this prior knowledge.
Method Input Size Depth Top-1 Top-5
Comparison with auto-generated networks - Our approach VGG [25] 224x224 16 28.5 9.90
achieves a significant improvement to the MetaQNN [2],
Inception V1 [30] 224x224 22 27.8 10.10
and even better than NAS’s best model (i.e. NASv3 more
Inception V2 [14] 224x224 22 25.2 7.80
filters) [37] proposed by Google brain which needs an ex-
ResNet-50 [11] 224x224 50 24.7 7.80
pensive costs on time and GPU resources. As shown in Ta-
ResNet-152 [11] 224x224 152 23.0 6.70
ble 4, NAS trains the whole system on 800 GPUs in 28
Xception(our test) [4] 224x224 50 23.6 7.10
days while we only need 32 GPUs in 3 days to get state-
ResNext-101(64x4d) [35] 224x224 101 20.4 5.30
of-the-art performance.
Block-QNN-B, N=3 224x224 38 24.3 7.40
Transfer block from CIFAR-100 to CIFAR-10 - We trans-
Block-QNN-S, N=3 224x224 38 22.6 6.46
fer the top blocks learned from CIFAR-100 to CIFAR-10
dataset, all experiment settings are the same. As shown Table 5. Block-QNN’s results (single-crop error rate) compare
in Table 3, the blocks can also achieve state-of-the-art re- with modern methods on ImageNet-1K Dataset.
sults on CIFAR-10 dataset with 3.60% error rate that proved
Block-QNN networks have powerful transferable ability.
Analysis on network parameters - The networks generated fairly, and we will consider this in our future work to further
by our method might be complex with a large amount of pa- improve the performance.
rameters since we do not add any constraints during train- As far as we known, most previous works of automatic
ing. We further conduct an experiment on searching net- network generation did not report competitive result on
works with limited parameters and adaptive block num- large scale image classification datasets. With the con-
bers. We set the maximal parameter number as 10M and ception of block learning, we can transfer our architecture
obtain an optimal block (i.e. Block-QNN-S) which outper- learned in small datasets to big dataset like ImageNet task
forms NASv3 with less parameters, as shown in Fig. 8(d). easily. In the future experiments, we will try to apply the
In addition, when involving more filters in each convolu- generated blocks in other tasks such as object detection and
tional layer (e.g. from [32,64,128] to [80,160,320]), we can semantic segmentation.
achieve even better result (3.54%).
6. Conclusion
5.3. Transfer to ImageNet
In this paper, we show how to efficiently design high per-
To demonstrate the generalizability of our approach, we formance network blocks with Q-learning. We use a dis-
transfer the block structure learned from CIFAR to Ima- tributed asynchronous Q-learning framework and an early
geNet dataset. stop strategy focusing on fast block structures searching.
For the ImageNet task, we set block repeat number We applied the framework to automatic block generation
N = 3 and add more down sampling operation before for constructing good convolutional network. Our Block-
blocks, the filters for convolution layers in different level QNN networks outperform modern hand-crafted networks
blocks are [64,128,256,512]. We use the best blocks struc- as well as other auto-generated networks in image classi-
ture learned from CIFAR-100 directly without any fine- fication tasks. The best block structure which achieves a
tuning, and the generated network initialized with MSRA state-of-the-art performance on CIFAR can be transfer to
initialization as same as above. The experimental results the large-scale dataset ImageNet easily, and also yield a
are shown in Table 5. The network generated by our frame- competitive performance compared with best hand-crafted
work can get competitive result compared with other human networks. We show that searching with the block design
designed models. The recently proposed methods such as strategy can get more elegant and model explicable network
Xception [4] and ResNext [35] use special depth-wise con- architectures. In the future, we will continue to improve the
volution operation to reduce their total number of parame- proposed framework from different aspects, such as using
ters and to improve performance. In our work, we do not more powerful convolution layers and making the searching
use this new convolution operation, so it can’t be compared process faster. We will also try to search blocks with lim-
ited FLOPs and conduct experiments on other tasks such as 68

detection or segmentation.
67

Acknowledgments

Accuracy (%)
66
This work has been supported by the National Natural
Science Foundation of China (NSFC) Grants 61721004 and 65
Start Exploitation RS Top1
61633021. RS Top5
64
BlockQNN Top1
Appendix BlockQNN Top5
63
1 21 41 61 81 101 121 141 161
A. Efficiency of BlockQNN Iteration (batch)

We demonstrate the effectiveness of our proposed Block- Figure 10. Measuring the efficiency of BlockQNN to random
QNN on network architecture generation on the CIFAR-100 search (RS) for learning neural architectures. The x-axis measures
dataset as compared to random search given an equivalent the training iterations (batch size is 64), i.e. total number of archi-
amount of training iterations, i.e. number of sampled net- tectures sampled, and the y-axis is the early stop performance after
works. We define the effectiveness of a network architec- 12 epochs on CIFAR-100 training. Each pair of curves measures
ture auto-generation algorithm as the increase in top auto- the mean accuracy across top ranking models generated by each
generated network performance from the initial random ex- algorithm. Best viewed in color.
ploration to exploitation, since we aim to getting optimal
auto-generated network instead of promoting the average
66
performance.
Figure 10 shows the performance of BlockQNN and ran- 65
dom search (RS) for a complete training process, i.e. sam-
pling 11, 392 blocks in total. We can find that the best model
Accuracy (%)

64
generated by BlockQNN is markedly better than the best
model found by RS by over 1% in the exploitation phase 63
RS-L Top1
on CIFAR-100 dataset. We observe this in the mean perfor- Start Exploitation
RS-L Top5
mance of the top-5 models generated by BlockQNN com- 62
BlockQNN-L Top1
pares to RS. Note that the compared random search method BlockQNN-L Top5
61
start from the same exploration phase as BlockQNN for 1 21 41 61 81 101 121 141 161
fairness. Iteration (batch)
Figure 11 shows the performance of BlockQNN
with limited parameters and adaptive block numbers Figure 11. Measuring the efficiency of BlockQNN with limited
(BlockQNN-L) and random search with limited parameters parameters and adaptive block numbers (BlockQNN-L) to ran-
dom search with limited parameters and adaptive block numbers
and adaptive block numbers (RS-L) for a complete training
(RS-L) for learning neural architectures. The x-axis measures the
process. We can see the same phenomenon, BlockQNN- training iterations (batch size is 64), i.e. total number of architec-
L outperform RS-L by over 1% in the exploitation phase. tures sampled, and the y-axis is the early stop performance after
These results prove that our BlockQNN can learn to gener- 12 epochs on CIFAR-100 training. Each pair of curves measures
ate better network architectures rather than random search. the mean accuracy across top ranking models generated by each
algorithm. Best viewed in color.
B. Evolutionary Process of Auto-Generated
Blocks
ally increase and the block tend choose ”Concat” as the last
We sample the block structures with median perfor-
layer. And we can find that the short-cut connections and
mance generated by our approach in different stage, i.e. at
elemental add layers are common in the exploitation stage.
iteration [1, 30, 60, 90, 110, 130, 150, 170], to show the evo-
Additionally, blocks generated by BlockQNN-L have less
lutionary process. As illustrated in Figure 12 and Fig-
”Conv,5” layers, i.e. convolution layer with kernel size of 5,
ure 13, i.e. BlockQNN and BlockQNN-L respectively, the
since the limitation of the parameters.
block structures generated in the random exploration stage
is much simpler than the structures generated in the ex- These prove that our approach can learn the universal de-
ploitation stage. sign concepts for good network blocks. Compare to other
In the exploitation stage, the multi-branch structures ap- automatic network architecture design methods, our gener-
pear frequently. Note that the connection numbers is gradu- ated networks are more elegant and model explicable.
 Input

Input
Input
Conv,3 Input Conv,3 Conv,5 Conv,3 Conv,1
Input
Input
Input Conv,5 Conv,3
Conv,3 Conv,5 Conv,3 Conv,3 Conv,5
AvgP,3 AvgP,1

Conv,1 Conv,3 AvgP,3 AvgP,3 AvgP,3 Conv,3 Conv,5 Conv,3 AvgP,3 Conv,5 Conv,3 Add Conv,5
Input Conv,1 Conv,5
Conv,1 Concat Conv,1 AvgP,3
Conv,1 Concat Add Conv,5
Conv,3 Add

MaxP,3 Add Add Concat Concat Concat Concat Concat

Random Exploration Exploitation from epsilon=0.9 to epsilon=0.1

Figure 12. Evolutionary process of blocks generated by BlockQNN. We sample the block structures with median performance at iteration
[1, 30, 60, 90, 110, 130, 150, 170] to compare the difference between the blocks in the random exploration stage and the blocks in the
exploitation stage.

Input Input

Input
Input

Conv,5 AvgP,3 MaxP,3 Conv,3 Conv,1 Conv,3

Input Input
Input Input Conv,3
Add Conv,1 MaxP,3 Conv,1 Conv,3 Conv,3
Conv,5 Conv,3 Conv,3 Conv,3 Conv,1 Conv,1
AvgP,3 Conv,1
MaxP,3 Conv,1 Concat
Conv,5 AvgP,3 MaxP,3 Concat
MaxP,3 Concat
AvgP,3 Add
Conv,3 Conv,5 MaxP,1 AvgP,3 AvgP,3 Conv,1 Conv,5 Conv,3
Conv,1 Conv,5 AvgP,1

Add Concat Concat Concat Concat Concat Concat Concat

Random Exploration Exploitation from epsilon=0.9 to epsilon=0.1

Figure 13. Evolutionary process of blocks generated by BlockQNN with limited parameters and adaptive block numbers (BlockQNN-L).
We sample the block structures with median performance at iteration [1, 30, 60, 90, 110, 130, 150, 170] to compare the difference between
the blocks in the random exploration stage and the blocks in the exploitation stage.

C. Additional Experiment [5] J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao,

A. Senior, P. Tucker, K. Yang, Q. V. Le, et al. Large scale dis-
We also use BlockQNN to generate optimal model on tributed deep networks. In Advances in neural information
person key-points task. The training process is conducted processing systems, pages 1223–1231, 2012. 5
on MPII dataset, and then, we transfer the best model found [6] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-
in MPII to COCO challenge. It costs 5 days to complete Fei. Imagenet: A large-scale hierarchical image database.
the searching process. The auto-generated network for In Computer Vision and Pattern Recognition, 2009. CVPR
key-points task outperform the state-of-the-art hourglass 2 2009. IEEE Conference on, pages 248–255. IEEE, 2009. 2
stacks network, i.e. 70.5 AP compares to 70.1 AP on COCO [7] T. Domhan, J. T. Springenberg, and F. Hutter. Speeding up
validation dataset. automatic hyperparameter optimization of deep neural net-
works by extrapolation of learning curves. In IJCAI, pages
3460–3468, 2015. 2
References [8] K. He and J. Sun. Convolutional neural networks at con-
strained time cost. In Proceedings of the IEEE Conference
[1] M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, on Computer Vision and Pattern Recognition, pages 5353–
D. Pfau, T. Schaul, and N. de Freitas. Learning to learn by 5360, 2015. 5
gradient descent by gradient descent. In Advances in Neural
[9] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into
Information Processing Systems, pages 3981–3989, 2016. 3
rectifiers: Surpassing human-level performance on imagenet
[2] B. Baker, O. Gupta, N. Naik, and R. Raskar. Designing neu- classification. In Proceedings of the IEEE international con-
ral network architectures using reinforcement learning. In ference on computer vision, pages 1026–1034, 2015. 6
6th International Conference on Learning Representations, [10] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learn-
2017. 1, 2, 4, 5, 6, 7, 8 ing for image recognition. In Proceedings of the IEEE con-
[3] J. S. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. Al- ference on computer vision and pattern recognition, pages
gorithms for hyper-parameter optimization. In Advances in 770–778, 2016. 1, 2, 7
Neural Information Processing Systems, pages 2546–2554, [11] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in
2011. 3 deep residual networks. In European Conference on Com-
[4] F. Chollet. Xception: Deep learning with depthwise separa- puter Vision, pages 630–645. Springer, 2016. 1, 2, 3, 7, 8
ble convolutions. In Proceedings of the IEEE conference on [12] S. Hochreiter, A. S. Younger, and P. R. Conwell. Learning to
computer vision and pattern recognition, 2017. 8 learn using gradient descent. In International Conference on
 Artificial Neural Networks, pages 87–94. Springer, 2001. 3 [29] R. S. Sutton and A. G. Barto. Reinforcement learning: An
[13] G. Huang, Z. Liu, K. Q. Weinberger, and L. van der Maaten. introduction, volume 1. MIT press Cambridge, 1998. 5
Densely connected convolutional networks. In Proceed- [30] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed,
ings of the IEEE conference on computer vision and pattern D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich.
recognition, 2017. 7, 8 Going deeper with convolutions. In Proc. IEEE Conference
[14] S. Ioffe and C. Szegedy. Batch normalization: Accelerating on Computer Vision and Pattern Recognition, pages 1–9,
deep network training by reducing internal covariate shift. In 2015. 1, 2, 8
International Conference on Machine Learning, pages 448– [31] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna.
456, 2015. 1, 2, 8 Rethinking the inception architecture for computer vision.
[15] D. Kingma and J. Ba. Adam: A method for stochastic opti- In Proceedings of the IEEE Conference on Computer Vision
mization. In 3rd International Conference for Learning Rep- and Pattern Recognition, pages 2818–2826, 2016. 1, 2
resentations, 2015. 6 [32] R. Vilalta and Y. Drissi. A perspective view and survey of
[16] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet meta-learning. Artificial Intelligence Review, 18(2):77–95,
classification with deep convolutional neural networks. In 2002. 3
Proc. Advances in Neural Information Processing Systems, [33] C. J. C. H. Watkins. Learning from delayed rewards. PhD
pages 1097–1105, 2012. 1 thesis, King’s College, Cambridge, 1989. 1
[17] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, [34] L. Xie and A. Yuille. Genetic cnn. In Proceedings of the
521(7553):436–444, 2015. 1 International Conference on Computer Vision, 2017. 2
[18] M. Li, L. Zhou, Z. Yang, A. Li, F. Xia, D. G. Andersen, and [35] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated
A. Smola. Parameter server for distributed machine learning. residual transformations for deep neural networks. In 2017
In Big Learning NIPS Workshop, volume 6, page 2, 2013. 5 IEEE Conference on Computer Vision and Pattern Recogni-
[19] L.-J. Lin. Reinforcement learning for robots using neural tion (CVPR), pages 5987–5995. IEEE, 2017. 8
networks. Technical report, Carnegie-Mellon Univ Pitts- [36] S. Zagoruyko and N. Komodakis. Wide residual networks.
burgh PA School of Computer Science, 1993. 1, 6 In British Machine Vision Conference, 2016. 7
[20] M. Lin, Q. Chen, and S. Yan. Network in network. In In- [37] B. Zoph and Q. V. Le. Neural architecture search with re-
ternational Conference on Learning Representations, 2013. inforcement learning. In 6th International Conference on
4 Learning Representations, 2017. 1, 2, 5, 7, 8
[21] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness,
M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland,
G. Ostrovski, et al. Human-level control through deep rein-
forcement learning. Nature, 518(7540):529–533, 2015. 1,
6
[22] A. Y. Ng, D. Harada, and S. Russell. Policy invariance under
reward transformations: Theory and application to reward
shaping. In ICML, volume 99, pages 278–287, 1999. 5
[23] S. Saxena and J. Verbeek. Convolutional neural fabrics. In
Advances in Neural Information Processing Systems, pages
4053–4061, 2016. 2
[24] J. D. Schaffer, D. Whitley, and L. J. Eshelman. Combinations
of genetic algorithms and neural networks: A survey of the
state of the art. In Combinations of Genetic Algorithms and
Neural Networks, 1992., COGANN-92. International Work-
shop on, pages 1–37. IEEE, 1992. 2
[25] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. In 3rd Interna-
tional Conference for Learning Representations, 2015. 1, 7,
8
[26] K. O. Stanley, D. B. D’Ambrosio, and J. Gauci. A
hypercube-based encoding for evolving large-scale neural
networks. Artificial life, 15(2):185–212, 2009. 2
[27] K. O. Stanley and R. Miikkulainen. Evolving neural net-
works through augmenting topologies. Evolutionary compu-
tation, 10(2):99–127, 2002. 2
[28] M. Suganuma, S. Shirakawa, and T. Nagao. A genetic pro-
gramming approach to designing convolutional neural net-
work architectures. In Proceedings of the Genetic and Evo-
lutionary Computation Conference, pages 497–504, 2017. 2