DeepTPI: Test Point Insertion with Deep

Reinforcement Learning
Zhengyuan Shi∗ § , Min Li∗ § , Sadaf Khan∗ , Liuzheng Wang† , Naixing Wang† , Yu Huang† and Qiang Xu∗
∗ Department of Computer Science and Engineering, The Chinese University of Hong Kong, Shatin, Hong Kong S.A.R.
{zyshi21, mli, skhan, qxu}@cse.cuhk.edu.hk
† HiSilicon Technologies Co., Ltd., China

Abstract—Test point insertion (TPI) is a widely used technique measurement approaches estimate the testability by fast ap-
for testability enhancement, especially for logic built-in self-test proximation and insert TPs to improve testability [7]–[9].
arXiv:2206.06975v2 [cs.LG] 27 Jun 2022

(LBIST) due to its relatively low fault coverage. In this paper, However, both of them are often ineffective on modern cir-
we propose a novel TPI approach based on deep reinforcement
learning (DRL), named DeepTPI. Unlike previous learning- cuits. The fault simulation approach requires the unacceptable
based solutions that formulate the TPI task as a supervised- runtime proportional to the size of the circuit [10], while the
learning problem, we train a novel DRL agent, instantiated accuracy of approximate measurement usually degrades due
as the combination of a graph neural network (GNN) and a to the reconvergence structures [11].
Deep Q-Learning network (DQN), to maximize the test coverage With the emergence of machine learning, various learning-
improvement. Specifically, we model circuits as directed graphs
and design a graph-based value network to estimate the action based approaches [12]–[14] achieve comparable test results
values for inserting different test points. The policy of the DRL to the previous work. [12] predicts the test coverage im-
agent is defined as selecting the action with the maximum value. provement of different TPs by an artificial neural network
Moreover, we apply the general node embeddings from a pre- (ANN). After that, [13] estimates the TP impact on delay
trained model to enhance node features, and propose a dedicated fault coverage improvement. However, the label generation
testability-aware attention mechanism for the value network.
Experimental results on circuits with various scales show that relies on the exhaustive fault simulation after inserting all
DeepTPI significantly improves test coverage compared to the possible TPs, which is expensive and intractable for large-
commercial DFT tool. The code of this work is available at scale circuits. [14] proposes a graph convolutional network
https://github.com/cure-lab/DeepTPI. (GCN) to classify nodes as either hard-to-observe or easy-to-
observe. The application of GCN to circuits is shown to be
I. I NTRODUCTION effective for automatic test pattern generation (ATPG) TPI.
Logic built-in-self-test (LBIST) [1] is a commonly used Nevertheless, the labels used for model training may not be
technique for in-field tests, especially for many safety-critical always accurate because they are obtained from commercial
scenarios [2], such as self-driving cars, healthcare systems, DFT tools, which in turn adopt heuristic algorithms. In theory,
and aircraft operating systems that have strict reliability re- the model performance is bounded by the heuristic algorithm
quirements. LBIST uses a pseudo-random pattern generator and is sub-optimal. To summarize, the supervised learning
to simulate the circuit behavior and verifies the corresponding approach is not fully suitable for the TPI problem.
responses. While LBIST demands low hardware and compu- Instead of collecting supervision for all TP candidates from
tational overhead, it fails to detect the random pattern resistant the DFT tools, in this work, we propose a novel learning-
(RPR) faults [1]. The RPR fault commonly occurs in complex based TPI technique, namely DeepTPI, which formulates the
modern circuits, where the gate has many fan-in wires. In TPI problem as a reinforcement learning (RL) task. The
most cases, there is only one valid input assignment out of optimization objective of DeepTPI is to maximize LBIST test
the quadratic fan-in number of total random combinations. coverage by inserting TPs sequentially. Compared with the
Therefore, the existence of the RPR faults results in the prior supervised-learning based TPI approaches, the benefits
insufficient test coverage under pseudo-random test. of our proposed solution are as follows: 1) The RL-based
To improve the low test coverage of LBIST caused by RPR approach is more sample-efficient than supervised learning and
faults, test point insertion (TPI) [3] has been studied exten- is free from the dependence on the vast number of training
sively, which inserts extra gates, i.e., test points (TPs), into samples; 2) The RL-based approach can fundamentally break
the circuit during the design-for-test (DFT) phase. As finding the upper bound of performance caused by supervision.
an optimal TP position is a known NP-hard problem [4], many In the DeepTPI formulation, we transform the circuit netlist
methods have been proposed to build computationally feasi- into a graph to naturally represent the circuit. The RL action
ble TPI algorithms. The straight-forward approaches perform space is defined as inserting a TP following a certain gate over
fault simulation to identify the RPR faults and then insert the circuit graph. With the node-level discrete action space,
TPs targeting on such faults [3], [5], [6]. The approximate- our RL agent is trained with Deep Q-learning [15], and is
equipped with a novel graph-based deep Q-network (Graph-
§ Both authors contributed equally to this research. DQN) to fit the action-value function. After aggregating the
 Sub Sub Sub Sub Sub Sub
OPs change circuit observability by inserting pseudo primary
Circuit Circuit Circuit Circuit Circuit Circuit output to the gate outputs to make faulty values on lines more
TP_EN TP_EN OP
observable [19].
Although TPI has various applications in different scenarios,
Fig. 1. Examples of Test Point Insertion(Left: AND Control Point, Middle: e.g., ATPG [20] or analog testing [21], this work addresses TPI
OR Control Point, Right: Observation Point) in the context of LBIST. On the one hand, LBIST generates
test patterns and verifies the corresponding responses on-
chip. Thus, as an in-field test technique, LBIST is in demand
message from neighbors for multiple iterations, the hidden
for high-assurance applications, such as implantable medical
state of each node is read out to predict the expected total
devices and self-driving vehicles. On the other hand, LBIST
reward of all possible actions for this node. We set the test
quality is more suffered from the presence of RPR faults than
coverage improvement as the reward in this TPI-for-LBIST
the deterministic testing [1]. Handling the RPR faults by TPI
task. The RL policy is defined as selecting the action with the
is known as a NP-hard problem [4]. The current heuristic TPI
maximum expected value.We also enhance the representation
approaches are still inefficient [2] and even decrease the circuit
ability of our Graph-DQN by involving prior knowledge from
testability in some cases [11].
DeepGate [16]. In addition, we analyze the dominant factors
Since the performance of most conventional TPI approaches
of testability and mimic testability by a customized attention
degrades for the modern circuits with complex structures,
mechanism. During GNN aggregation, the testability-aware
various learning-based approaches are proposed to select TPs
attention mechanism gives higher weight to those nodes with
using advanced machine learning algorithms. [12] and [13]
more significant impacts on testability.
estimate the quality of TPs with an artificial neural network
The contributions of this work are summarized as follows:
(ANN), but the model relies on large amounts of samples.
• We formulate the TPI problem as a reinforcement learn-
Unlike supervised learning approaches, the RL algorithm used
ing task and train an RL agent with the Deep Q-learning in this paper trains an agent by action-environment interaction
algorithm. To the best of our knowledge, this is the rather than exhaustively executing all possible TP candidates.
first work applying reinforcement learning on test point [14] classifies each node into easy-to-observe or hard-to-
insertion. observe. However, the binary labels are collected from a
• We transform the RL state into a graph and deploy
heuristic testability analysis algorithm, which is sub-optimal
a dedicated graph-based deep Q-network (Graph-DQN) and bounds the performance of the resulted model in theory.
to assist RL decisions. Moreover, the general neural On the contrary, our proposed RL-based approach is not
representation from pre-trained DeepGate [16] is embed- limited by the upper-bound performance.
ded into the node feature to involve the prior circuit
knowledge. B. Reinforcement Learning
• We deploy a dedicated GNN to assist RL policy deci- Recently, reinforcement learning (RL) has achieved out-
sions. By analyzing the essence of testability, we propose standing performance in different applications. In general, RL
a testability-aware attention mechanism in the GNN to is suitable for solving the problems without apparent supervi-
capture more useful information during aggregation for sion, which is often the case in many EDA applications. For
the TPI task. example, getting the optimal test patterns in software-based
We organize the remaining sections as follows. Some related self-test (SBST) is impractical. Thus, [22] formulates SBST as
work about this paper is summarized in Section II. Section III an RL problem and rewards the agent when a newly detected
models TPI as an RL problem and introduces the RL com- fault is reached. Besides, the best quality of result (QoR) is
ponents. The architecture and methodology of DeepTPI are hard to be produced as supervision in logic synthesis due to
explained in Section IV. The experimental results are shown the exponential number of possible optimization permutations.
in Section V. Finally, Section VI concludes this paper. [23], [24] convert logic synthesis as an RL problem, whose
optimization target is set as reducing area or depth.
II. R ELATED W ORK
In EDA, the most natural representation of circuits, interme-
A. Test Point Insertion diate RTL, netlists, and layouts are graphs [25]. Consequently,
Test point insertion [3] is a circuit modification technique a few RL approaches incorporate graph neural networks as
that adds extra gates into circuits during the design-for-test the backbones to capture the graph structures. For instance,
(DFT) phase and enhances circuit testability. As shown in GCN-RL circuit designer [26] embeds the topology graph
Fig. 1, the test points are categorized into two types: control information with GNN and uses an RL optimization algorithm
points (CPs) and observation points (OPs). CPs are typically to achieve the high Figures of Merit (FoM) for automatic
implemented as OR gates for control-1 TPs or AND gates for transistor sizing problems. [27] proposes a GNN model to
control-0 TPs. During the testing, a test enable pin forces lines encode the information about netlist and make macro place-
to their controlled values [17]. The goal of control CPs is to ment decisions with another policy network. Besides, [28] and
increase the circuit controllability, i.e., the probability of acti- [29] combine the RL agent with GNN to design combinatorial
vating faults and sensitizing the fault propagation paths [18]. optimization solvers. Our proposed DeepTPI shares the similar
 Environment

……
Circuit Insert TP Circuit Insert TP Circuit
t=0 t=0 t=1 t = T-1 t=T
Update
State
Netlist Transition Netlist
Fig. 2. Test Point Insertion as an Markov Decision Process (MDP) st st+1
F(st, at)

Action at
intuition with these works, which learns the circuit knowledge Reward rt
via a GNN and conducts TPI actions based on the learned
graph embeddings.
DeepTPI Agent
III. R EINFORCEMENT L EARNING F ORMULATION
In this work, we target the test point insertion (TPI) prob- Fig. 3. RL Agent - Environment Interaction in DeepTPI
lem, in which the objective is to improve the test coverage
under LBIST random patterns by inserting control points which maximizes the expected cumulative rewards. The MDP
and/or observation points. Such a sequential decision-making also defines a discounting factor γ 0 , which determine the
problem can be modeled as a Markov Decision Process different weights on long-term reward and short-term reward.
(MDP). Moreover, an optimal decision strategy in such a The overall objective of the agent decision is to maximize
sequential decision-making setting can be identified by rein- PT −1
the expected discounted reward: G = E[ t=0 γ t rt ], where
forcement learning (RL) methods [30]. Hence, in this section, γ t = Power(γ 0 , t).
we formally introduce the TPI problem from a reinforcement In the TPI setting, the trained agent shall select the optimal
learning perspective, and build the connections between key positions and insert the corresponding test points so that
elements of RL and TPI. the maximum test coverage improvement is achieved. We
Given a circuit netlist, we form a directed graph G = (V, E), instantiate the reward function R(st , at , st+1 ) in Eq. (1) as
where each gate v in the circuit is a vertex belong to V, and test coverage improvement (TC Imp.) after inserting T TPs in
e(u, v) ∈ E is a directed edge from node u to node v. We total, that is the difference of test coverage between the final
use T ∈ N to denote the time horizon, i.e., the number of test circuit with T TPs inserted and the initial circuit. The reward
points to be inserted. The circuit after inserting t test points function belongs to the terminal reward, which is nonzero only
can be represented as G t . Equipped with the above notations, after inserting T TPs. Thus, we can set the discounting factor γ
we then formulate the TPI problem as an MDP (Shown in to an arbitrary value in [0, 1) because the long-term or short-
Fig. 2), which consists of the following four key elements: term discounting does not affect the optimization objective.
• State ST : the set of states. In TPI task, the state is Ideally, the trained RL agent selects a list of TPs with the size
reflected as circuit with graph structure. The state is of T to achieve globally optimal test coverage improvement.
represented as the circuit at time step t: st := G t ∈ ST .
• Action A: the set of actions that an agent can execute,
(
0, t = 0, · · · , T − 2,
i.e., inserting test points into the circuit. The action rt = R(st , at , st+1 ) = (1)
corresponds to inserting one kind of TPs on the follow- T C Imp., t = T − 1
ing wire of a particular gate, which can be defined as To approach the described objective above, we designate
at := (v, type) ∈ A, wherein v is the selected gate for the a combination of graph neural network and deep Q-learning
test point insertion and type ∈ {AND-CP, OR-CP, OP} that approximates an optimal action-value function for TPI.
indicates what type of test point to be inserted. We discuss the methodology in detail in the following section.
t
• State transition function F: Given state s and an action
a at time t, the function produces the updated state st+1 .
t IV. M ETHODOLOGY
Since the the circuit after an action of TPI is uniquely A. Overview of DeepTPI
determined, the state transition function is deterministic. Fig. 4 shows the overview of the DeepTPI RL agent. Firstly,
We denote the transition function as st+1 = F(st , at ). the input circuits are preprocessed into a unified format along
t t t t+1
• Reward function R: step reward r = R(s , a , s ) with the rich node features extracted from a pre-trained model
t
caused by the corresponding action a with state tran- (i.e., DeepGate [16]). More precisely, given a pool of circuit
sition from st to st+1 . In TPI task, we consider the netlists for training, we process them via an equivalent trans-
improvement of the test coverage as the reward. formation into And-Inverter Graph (AIG), wherein only three
As shown in Fig. 3, at each step t, the agent extracts gate types, i.e., primary input (PI), AND and NOT. The details
information from the current state st and selects an action of such transformation are elaborated in Sec. IV-B. Besides the
at . Then, the circuit inside the environment is updated by circuit transformation, we present a feature-based approach to
the state transition function st+1 = F(st , at ). The primary enrich the node features. We extract the node embeddings for
target to solve the MDP problem is to train an RL agent each gate from a pre-trained DeepGate model, which learns
with optimal policy π, a mapping that outputs an action the general-purpose gate embeddings (See Sec. IV-C).
 Iteration 0
ℎ!"

ℎ#"
5

argmax
x5 Iteration 1 ℎ$" q(0, OP) q(5, OP)
DeepGate

……
3 4

Action
MLP
x3 x4 ℎ%" q(0, AND) q(5, AND)
…… ℎ&" q(0, OR) q(5, OR)
01 2 x0 x1 x2
Iteration D-1
ℎ'"

AIG Circuit Graph with rich Node Embedding Expected Value

node feature Graph Neural Network Readout Layer

Graph-DQN Value Network

Fig. 4. The Overview of DeepTPI Agent

Secondly, since the TPI procedure can be formulated as an Algorithm 1: DeepTPI Application Process
MDP with a discrete action space as discussed in Sec. III, the Input: The pre-TPI circuit ckt, Trained value network
value-based deep Q-learning [15] algorithm can be applied for Graph-DQN and Pre-trained model DeepGate
0
this task. We design a Graph-based Deep Q-Network (Graph- Output: The post-TPI circuit ckt
DQN), instantiated as a GNN along with a readout layer. The /* Initialize circuit graph G = (V, E)
Graph-DQN consumes the input graphs and learns to capture */
the testability properties of a logic gate in the context of the 0
1 G = ConvertAIGGraph(ckt);
whole circuit (See Sec. IV-D). To better embed testability /* RL decision */
by the GNN model, we analyze the requirements of faults 2 Define TP list `;
being detectable and propose a novel testability-aware atten- 3 Define time-step t = 0;
tion mechanism in the aggregation function. The customized 4 while t ≤ number of TPs do
attention-based aggregation function enables differentiation of /* Calculate the initial feature */
predecessors and successors according to their contribution 5 node embt = DeepGate(G t );
to fault sensitization and fault propagation, respectively (See 6 Xt = Concatenate(node embt , OneHot(V t )) ;
Sec. IV-E). /* Calculate expected value */
Finally, given expected values for all actions predicted 7 Qt = Graph-DQN(G t , Xt );
by Graph-DQN, the RL agent selects the action with the /* Choose action */
maximum expected value to be conducted. Such pipeline is 8 at = GetAction( arg max(Qt ));
repeated under the interaction with the environment until a pre- 9 `.append(at );
defined termination criterion is satisfied (e.g., hit the maximum /* Transit state */
number of test points to be inserted). Alg. 1 provides the 10 G t+1 = F (G t , at );
pseudo-code of DeepTPI. 11 end
/* Generate the post-TPI circuit */
B. Unified AIG of Circuits 0
12 ckt = InsertTPs(ckt, `);
We opt for a unified AIG representation of circuits. Since the 13 return ckt0 ;
different circuits in the DFT phase are mapped into standard
cells from different libraries, the gate types and local structures
of the circuits are likely to have non-uniform distribution.
Such heterogeneity across circuit designs is a challenge for combinations without logic optimization. Moreover, only those
GNN and RL model development. Taking the fact that any gates in AIG that have the corresponding positions in original
type of logic gate can be converted into the combination of circuits are considered TP candidates. Fig. 5 shows a toy
AND gate and Inverter (NOT gate), we map circuits to their example of the applied transformation. Only A, B, C, D, E
corresponding AIG counterparts. It should be noted that the are considered as the candidate positions for the RL agent.
transformation process does not rely on the AIG synthesis D1 , D2 , D3 are masked out because these nodes in AIG format
tool [31] because test point insertion in the DFT stage prohibits do not correspond to any gates in the original circuit.
modifying the circuit structure except for TPs. Instead, we There are two benefits of the AIG transformation: 1) we map
simply map the original gates to their equivalent And-Inverter the gate into the combination of AND and NOT gates directly
 A D D1 The optimal policy π is defined as executing the action with
A D3 D
B E maximum value:
E B
D2
C C
at = π(st ) = arg max
t
(Q(st , at )) (2)
a
Fig. 5. Convert Circuit to AIG
In TPI task, since the state st solely depends on a di-
rected acyclic graph G = (V, E), the action space is defined
rather than optimizing the logic so that the transformation is of
at node-level over an arbitrary graph. To accommodate the
linear complexity and can scale on large circuits; 2) including
requirements of dealing with graph-based data, we refer to a
PI, only three types of nodes are considered, which would
graph neural network, and implement the value network as the
dramatically reduce the difficulty of RL training.
graph-based deep Q-Network (Graph-DQN) to approximate
C. Features of Logic Gates from DeepGate the action value. The whole architecture is shown as the
Graph-DQN value network block in Fig. 4. We calculate
A natural way to assign the node features for logic gates is
the node embedding for each node for multiple iterations by
to use the one-hot encoding of gate types. When PIs are taken
a GNN and readout the expected value with a multi-layer
into account, there are three gate types in AIG: P I, AND and
perceptron (MLP).
NOT. Even though such one-hot encoding is a widely adopted
convention for graph learning [32], [33], it cannot include any We formally introduce the GNN structure used in
specific properties of logic gates to benefit RL decision, such Graph-DQN. The GNN propagates the message between each
as the gate behaviors and testability. With limited information node vi ∈ V and its neighbors N (i) with two functions:
on logic gates to utilize, the RL model may struggle to learn aggregation function AGGR and update function UPDATE.
and exposes the RL agent to a high risk of corruption. The aggregation function enables the message-passing of all
To augment the semantic information embedded in node fea- nodes with arbitrary degrees. The update function combines
tures, we follow the success of model pre-training techniques the node feature with the incoming message and updates
in deep learning community [34], [35], and adopt a feature- the hidden state iteratively. We apply gated recurrent neural
based approach, i.e., using the pre-trained node embeddings (GRU) [37] to implement the UPDATE function. We define
as the additional features to incorporate the gate semantics. the GNN computations as Eq. 3, where hdi is the hidden
To be specific, we extract the node embeddings from a pre- state and d = 0, 1, ..., D − 1 is the number of iterations.
trained DeepGate model [16] and feed them as the initial The initial hidden state h0i is assigned to node feature xi .
node features for RL training. The task of DeepGate is to The aggregation function can be implemented by average
approximate the simulated probability, i.e., the probability pooling [38] or linear transformation [39], but these methods
of the gate being logic 1 under random logic simulation. do not differentiate the neighbors who dominate the testability.
With strong biases introduced into DeepGate design, the node To mimic the essence of testability, we borrow the attention-
embeddings for logic gates embed both logic function and based aggregation function from [40]. We elaborate on the
structural information of a circuit. We concatenate the node testability-aware attention mechanism in Sec. IV-E.
embeddings from pre-trained DeepGate and the 3-d one-hot
encoding of gate types together to construct the node feature
xi for node vi . The effectiveness of the proposed feature-based hd+1
i = UPDATEd+1 (AGGRd+1 ({hd+1 k kk ∈ N (i)}), hdi )
pre-training is demonstrated by the ablation study in Sec. V-E. (3)
The GNN message-passing process performs for D itera-
D. Graph-based Deep Q-Network
tions. The hD
i after the node embedding of the final iteration
In the MDP formulation described in Sec. III, we define d = D − 1 is fed into the readout function READOUT to
the action as inserting a TP following candidate gate, and the produce the expected total value of all possible actions:
state transition as modifying the circuit with the newly inserted
TP. The Q-learning algorithm is suitable to explore an optimal qi = READOUT(hD
i ) (4)
policy for such MDP problem with a discrete gate-level action
space and deterministic state transition [36]. Therefore, we
where qi is a three-dimension vector for each node vi ∈ V
apply Q-learning to RL-formulated TPI problem.
that presents the action value when inserting the AND control
The classical Q-learning approximates the expected reward
point, OR control point, or observation point. The RL agent
for all possible actions based on the current state, and main-
chooses the action with the maximal value to maximize the
tains an action-value function as a table (also named Q-
total reward.
Table). Due to the increasing scale of action space in practice,
We define the action selection process as following four
the deep Q-learning [15] introduces a value network (also
steps:
named Q-Network) to implicitly fit the action-value function.
Specifically, at each time step t, the Q-Network (Q) predicts 1) Node feature construction: The initial node feature Xt is
the future rewards of all possible actions at in current state st . concatenated of the node embedding node embt from
 the pre-trained DeepGate model and one-hot gate type D
encoding of each node OneHot(V t ).
0/0 0/1 C1
node embt = DeepGate(G t ) A C
t t
X = Concatenate(node emb , OneHot(V )) t B C2
SA1 E
(5) 0/0
0/1
2) Value approximation: Based on the initial node feature 1
Xt and graph G t , the proposed Graph-DQN predicts the
Fig. 6. An Example of Stuck-at-1 Fault
expected cumulative reward Qt for each possible action
in node-level.
Qt = Graph-DQN(G t , Xt ) (6) probability to be logic 1. Moreover, condition (b) requires
3) Action execution: The agent policy is selecting the action the fault-effect being observed, no matter on which primary
with the maximum expected reward. output. Take the fault-effect 0/1 propagation in Fig. 6 as
an example, there are two paths following stem gate C.
at = GetAction(arg max(Qt )) (7) Sensitizing the path through E by forcing one line to be
4) State transition: The circuit structure changes after in- 1 is easier than D. The probability of observing this fault-
serting the corresponding TP. The environment updates effect relies only on the difficulty of sensitizing the path
the state to be G t+1 . through successor E. Therefore, the neighbor that can easily
propagate fault-effect is more noteworthy. To summarize, the
G t+1 = F (G t , at ) (8) predecessors and successors play different and significant roles
in the testability of each gate.
Besides, in the RL training process, the parameter θ in
Graph-DQNθ is updated iteratively by: To mimic the above properties during GNN propagation,
we turn to graph attention network [40]. Graph attention
L(θ) = Loss(Graph-DQNθ (st , at ), R(st , at , st+1 )+ network computes the hidden representations of each node in
0 (9)
γ 0 argmaxat+1 (Graph-DQNθ̂ (st+1 , at+1 ))) the graph by attending over its neighbors following a attention
0
mechanism. Inspired by this work, our RL value network also
where the target network Graph-DQNθ̂ is another value net- follows an attention strategy. The Graph-DQN distinguishes
work used as a measure of ground truth (i.e. the expected the attention on predecessors and successors because the pre-
future reward) and stabilize learning. The parameter θ̂ is decessors determine whether the fault-effect can be activated,
updated to θ every U epochs. We use Mean-Squared (MSE) and the successors affect the effect observation. Therefore,
loss as the loss function Loss. we define the testability-aware attention mechanism in the
aggregation function AGGR as below:
E. Testability-Aware Attention Mechanism
In the logic circuits, when the logic gate receives a control- X
ling value from one of the inputs, the output value is almost mdi(pre) = d
α(i,p) hdp ,
independent of the other inputs [41]. For example, the fan-in p∈P(i)
value 0 is a controlling value of AND gate and forces the d
where α(i,p) = sof tmax(w0> hd−1
i
>
+ wpre hdp )
gate to be 0 no matter what the values of the other fan-in p∈P(i)
X (10)
gates are. The rule can be derived from the other types of mdi(suc) = d
β(i,s) hds ,
gates. Such similar gate property plays an important role in s∈S(i)
testability analysis. Fig. 6 shows an example of detecting a d
where β(i,s) = sof tmax(w0> hid−1 + wsuc
>
hds )
stuck-at-1 (SA1) fault at gate C, where 0/1 means the expected s∈S(i)
logic value of this line is 0, but the fault causes the value to
1. The circuit must fulfill two conditions to enable such SA1
fault to be detected: (a) at least one random pattern assigning wherein P(i) and S(i) are the predecessors and successors of
logic 0 at the gate C, i.e., the fault-effect (logic difference) node i, respectively. During the training process, two indepen-
is activated; (b) at least one path that allows the fault-effect dent model parameters, wpre and wsuc , are used for calculating
d d
to propagate from C to any primary outputs E or D, i.e., the the attention weighting coefficients α(i,p) and β(i,s) . These two
fault-effect can be observed. We assume the testability as the coefficients represent the appropriate attention on aggregating
probability of a gate meeting both conditions (a) and (b). information from different predecessors or successors.
The predecessor with controlling value dominates the fault After we obtain the aggregated message mdi(pre) and
activation. For example, if there is a controlling value (logic mdi(suc) , we update them together as the output of AGGR
1) in any fan-in wire of gate C, the fault cannot be activated. function. With the testability-aware attention mechanism, the
Intuitively, to estimate the probability of meeting the above GNN can generate the embedding with much more testability
condition (a), we need focus on the fan-in gates with a high information for RL decisions.
 TABLE I TABLE II
T HE S TATISTICS OF T RAINING C IRCUITS T HE S TATISTICS OF T ESTING C IRCUITS IN AIG FORMAT

# Gates # PIs # POs # Levels TC Name # Gates # PIs # POs # Levels TC

Min. 83 8 1 7 24.08% D1 b15 C 27,838 485 449 158 90.61%
Max. 656 98 24 80 100.00% D2 b20 C 61,667 522 507 165 90.60%
Avg. 201.73 23.87 7.42 15.77 90.59% D3 b21 C 62,817 522 507 167 89.73%
D4 b22 C 91,505 767 750 173 91.70%
D5 i2c 3,400 136 127 34 86.11%
D6 max 9,439 512 129 333 52.65%
V. E XPERIMENTS D7 b17 C 99,466 1,452 1443 234 86.84%
D8 mem ctrl 127,353 1,028 967 195 69.95%
In this section, we present our constructed circuit datasets
for training and testing, as well as the baseline in Sec. V-A.
We elaborate the model settings in Sec. V-B. Then, we TABLE III
discuss the performance comparison between DeepTPI and the T HE M ODEL C OMPLEXITY OF G RAPH -DQN AND D EEP G ATE
commercial DFT tool, and analyze the advantage of DeepTPI
Graph-DQN DeepGate Total
in Sec. V-C. To explore the effectiveness of the testability-
aware attention mechanism and the feature-based pre-training, # FLOPs 116.47 M 60.63 M 177.10 M
# Param. 59.01 K 69.13 K 128.14 K
we present two ablation studies in Sec. V-D and Sec. V-E,
respectively. Finally, we analyze the computational complexity
and discuss some potential improvements in Sec. V-F.
B. Model Settings
A. Circuit Dataset and Baseline
In the Graph-DQN model configuration, we set the di-
We extract 400 sub-circuits from ISCAS89 [42] and mension of node hidden state as 64 and the iteration of
ITC99 [43], and convert them to AIG format for RL training. message passing as 10. We also involve the universal logic gate
Tab. I summarizes the statistics of training circuits. # Gates representation pre-trained model DeepGate. The pre-trained
denotes the number of gates in circuits, while # PIs and # DeepGate keeps the same configurations and hyperparameters
POs represent the number of primary inputs (PIs) and primary as the original specification [16]. We set the dimension of node
outputs (POs), respectively. # Levels is the total number hidden state as 64 and the iteration of hidden state update as
of logic levels. The test coverage (TC) on original circuits 10 for DeepGate. The DeepGate model is trained with 10,824
is reported by a commercial LBIST tool under the same subcircuits extracted from open-source benchmarks.
volume of random test patterns, i.e., 300K patterns. In some
The model complexity is illustrated in Tab. III, where
cases, especially for these circuits with high test coverage,
# FLOPs is the number of floating-point operations and
inserting the inappropriate test point may reduce the random
# Param. is the total number of trainable parameters. The RL
test coverage. For example, the extra AND control point on
agent is trained for 500 episodes on each circuit in Tab. II. We
the fan-in wires of the AND gates causes the SA1 faults more
use the ADAM optimizer with the learning rate lr = 0.0001.
hard to be activated. To explore the optimal policy avoiding
In the following experiment, we keep the same configuration
the test coverage loss for these circuits, our training dataset
for all DeepTPI agents.
also includes the circuits that can be fully tested by LBIST
(TC=100.00%).
C. Performance Comparison with Commercial DFT Tool
For inference, we choose eight large combinational cir-
cuits from two open-source benchmarks, i.e., ITC [43] and This section compares the TPI performance between the
EPFL [44], and compare the test coverage reported from state-of-the-art commercial DFT tool and our proposed
the state-of-the-art commercial DFT tool and DeepTPI. To DeepTPI. The user manual of the tool indicates that the
examine the generalization property of our RL approach, the testability analysis algorithm is based on COP [7] and pattern
size of testing circuits varies from 3.4K to 127.4K, which is simulation [45].
much larger than those in the training dataset. The statistics of To ensure fairness, we use the same pre-TPI circuits and
the testing circuits in AIG format are listed in Tab. II. It should assign the same number of inserted TPs. The random test pat-
be noted that there is no overlap between training circuits and terns are generated by the same commercial LBIST tool with
testing circuits. The pre-TPI test coverage is shown in Column an equivalent configuration. One group of post-TPI circuits is
TC in Tab. II. produced by the commercial tool and the other is generated by
According to practical experience, the inserted test points sequential actions from DeepTPI. The test coverage of these
for LBIST coverage improvement occupy 0.1%-2.0% extra two groups are abbreviated as Com. Tool TC. and DeepTPI
area over the entire circuit. Typically, there is a strong positive TC, respectively. As shown in Tab. IV, DeepTPI outperforms
correlation between the number of TPs and test coverage the commercial DFT tool significantly. For instance, our agent
improvement. In the following experiments, we fix the number can achieve 4.78% test coverage improvement (abbreviated as
of inserted test points (# TPs) as 1.0% of # Gates for all testing Imp.) on the testing circuit D3 higher than that 0.53% from
circuits. the commercial DFT tool, which reaches 9.02× improvement.
 TABLE IV TABLE VI
T HE T EST C OVERAGE I MPROVEMENT C OMPARISON BETWEEN D EEP TPI T HE T EST C OVERAGE I MPROVEMENT C OMPARISON BETWEEN W / O
AND C OMMERCIAL DFT T OOL P RE - TRAIN AND W / P RE - TRAIN

TC # TPs Com. Tool TC Imp. DeepTPI TC Imp. TC # TPs w/o Pre-Train Imp. w/ Pre-train Imp.
D1 90.61% 278 91.50% 0.89% 93.20% 2.59% D1 90.61% 278 91.63% 1.02% 93.20% 2.59%
D2 90.60% 616 91.32% 0.72% 95.02% 4.42% D2 90.60% 616 92.26% 1.66% 95.02% 4.42%
D3 89.73% 628 90.26% 0.53% 94.51% 4.78% D3 89.73% 628 91.42% 1.69% 94.51% 4.78%
D4 91.70% 915 92.44% 0.74% 95.59% 3.89% D4 91.70% 915 93.18% 1.48% 95.59% 3.89%
D5 86.11% 34 89.72% 3.61% 94.44% 8.33% D5 86.11% 34 89.23% 3.12% 94.44% 8.33%
D6 52.65% 94 58.34% 5.69% 63.01% 10.36% D6 52.65% 94 57.49% 4.84% 63.01% 10.36%
D7 86.84% 994 87.96% 1.12% 91.67% 4.83% D7 86.84% 994 90.52% 3.68% 91.67% 4.83%
D8 69.95% 1,273 72.34% 2.39% 76.98% 7.03% D8 69.95% 1,273 75.88% 5.93% 76.98% 7.03%
Avg. 1.96% 5.78% Avg. 2.93% 5.78%

TABLE V
T HE T EST C OVERAGE I MPROVEMENT C OMPARISON OF D IFFERENT between the nodes benefiting fault sensitizing and the nodes in
AGGREGATOR the easiest fault propagation path. As a result, the agent with
TC w/o Att. Imp. Nor. Att. Imp. DeepTPI Att. Imp.
testability-aware attention increases 2.06× TC improvement
than the agent without attention.
D1 90.61% 92.62% 2.01% 92.75% 2.14% 93.20% 2.59%
D2 90.60% 91.31% 0.71% 92.04% 1.44% 95.02% 4.42%
D3 89.73% 91.60% 1.87% 90.79% 1.06% 94.51% 4.78% E. Effectiveness of Feature-Based Pre-Training Approach
D4 91.70% 93.67% 1.97% 92.84% 1.14% 95.59% 3.89%
D5 86.11% 87.67% 1.56% 90.21% 4.10% 94.44% 8.33% To investigate the effectiveness of the feature-based pre-
D6 52.65% 58.29% 5.64% 59.45% 6.80% 63.01% 10.36% training scheme, we train an RL agent without the pre-trained
D7 86.84% 89.84% 3.00% 90.76% 3.92% 91.67% 4.83%
D8 69.95% 75.68% 5.73% 75.87% 5.92% 76.98% 7.03%
DeepGate model (abbreviated as w/o Pre-Train). Different
from including the node embedding by the pre-trained Deep-
Avg. 82.27% 2.81% 3.32% 5.78%
Gate model, the initial node feature for w/o Pre-Train only
contains the 3-d one-hot encoding of gate types. Tab. VI shows
the test coverage improvement (Imp.) by these two agents
On average, we can achieve 2.95× test coverage improvement with the same number of inserted TPs. As the customized
than the tool. agent (w/o Pre-Train) lacks the prior logic circuit knowledge,
D. Effectiveness of Testability-Aware Attention Mechanism it only improves 2.93% test coverage on average, which is
much lower than 5.78% Imp. obtained by the original agent
To demonstrate that our testability-aware attention mecha-
(w/ Pre-train).
nism can capture more helpful information in the TPI task, we
To conclude, the pre-trained model provides a more helpful
design an ablation study with different aggregation configura-
gate representation. Such abundant representation can help the
tions. We apply three different aggregators in our RL agent:
agent achieve about 1.97× test coverage improvement with the
the aggregator without attention (w/o Att.), the aggregator
same TPI overhead.
with normal logic-aware attention, same as [16], [33] (Nor.
Att.) and the aggregator with our proposed testability-aware
F. Analysis of Computational Complexity
attention as Sec. IV-E (DeepTPI Att.). Except for the above
aggregation function, the remaining RL components keep the The proposed DeepTPI utilizes modern GPUs for RL
same hyperparameters. training and inference. To give an idea of the computational
Tab. V shows the experimental results. The test coverage im- complexity, we take circuit D1 as an example. The DeepTPI
provement between post-TPI and the pre-TPI circuit is denoted takes 132.07 seconds to insert 10 TPs on D1 (running on
as Imp.. The RL agent without any attention mechanism only Nvidia GeForce GTX 1080 Ti GPU), while the commercial
achieves 2.81% Imp. on average. This setting performs worst DFT tool takes 12.79 seconds to insert the same number of
because it cannot determine the impact of detecting faults of TPs. Overall, DeepTPI takes around 10× more runtime than
different neighbors. Although normal attention can distinguish the commercial tool. As TPI is a one-time effort process in
the dominant neighbors, it cannot capture the synergistic the DFT phase, the computational overhead of DeepTPI is
effect on testability between predecessors and successors. The acceptable.
normal attention also has an inferior performance to our We also measure the runtime for different modules of
proposed testability-aware attention mechanism. The former DeepTPI. Fig. 7 shows the runtime breakdown of the DeepTPI
only improves 3.32% test coverage on average, lower than inference on the circuit D1. The pre-trained DeepGate and
5.78% by our proposed attention mechanism. Graph-DQN take 35.62% and 30.53% out of the total time,
To summarize, the testability-aware attention is more ef- respectively. Although the Graph-DQN is more complex than
ficient on the TPI task because it not only captures the DeepGate (see Tab. III), DeepGate consumes more time as it
dominance of neighbors but also considers the coordination updates the node hidden state with the orders of the logic level.
 5.00% [5] N. Tamarapalli and J. Rajski, “Constructive multi-phase test point inser-
35.62% tion for scan-based bist,” in Proceedings International Test Conference
28.85% 1996. Test and Design Validity. IEEE, 1996, pp. 649–658.
DeepGate [6] V. S. Iyengar and D. Brand, “Synthesis of pseudo-random pattern
testable designs,” in Proceedings.’Meeting the Tests of Time’., Interna-
GraphDQN tional Test Conference. IEEE, 1989, pp. 501–508.
[7] F. Brglez, “On testability of combinational networks,” 1984.
[8] N. A. Touba and E. J. McCluskey, “Test point insertion based on path
State Transition tracing,” in Proceedings of 14th VLSI Test Symposium. IEEE, 1996,
pp. 2–8.
[9] D. Das and N. A. Touba, “Reducing test data volume using external/lbist
Others hybrid test patterns,” in Proceedings International Test Conference 2000
(IEEE Cat. No. 00CH37159). IEEE, 2000, pp. 115–122.
30.53% [10] H.-C. Tsai, C.-J. Lin, S. Bhawmik, and K.-T. Cheng, “A hybrid algorithm
for test point selection for scan-based bist,” in Proceedings of the 34th
Fig. 7. The Runtime Breakdown of the DeepTPI Inference annual Design Automation Conference, 1997, pp. 478–483.
[11] S. Roy, B. Stiene, S. K. Millican, and V. D. Agrawal, “Improved
pseudo-random fault coverage through inversions: a study on test point
architectures,” Journal of Electronic Testing, vol. 36, no. 1, pp. 123–133,
Contrary to aggregating messages level-by-level, the Graph- 2020.
DQN updates all nodes over the whole graph simultaneously. [12] Y. Sun and S. Millican, “Test point insertion using artificial neural
Another time-consuming computation process is the State networks,” in 2019 IEEE Computer Society Annual Symposium on VLSI
(ISVLSI). IEEE, 2019, pp. 253–258.
Transition, which takes 28.8% out of total time. The State [13] S. Millican, Y. Sun, S. Roy, and V. Agrawal, “Applying neural networks
Transition refers to the re-calculation of both the forward logic to delay fault testing: Test point insertion and random circuit training,”
level (from PI to PO) and backward logic level (from PO to PI) in 2019 IEEE 28th Asian Test Symposium (ATS). IEEE, 2019, pp.
13–135.
after inserting one TP. It should be noted that maintaining logic [14] Y. Ma, H. Ren, B. Khailany, H. Sikka, L. Luo, K. Natarajan, and B. Yu,
level only profits the pre-trained model inference, as DeepGate “High performance graph convolutional networks with applications in
aggregates the message level-by-level. testability analysis,” in Proceedings of the 56th Annual Design Automa-
tion Conference 2019, 2019, pp. 1–6.
There are several potential ways to further improve the ef- [15] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G.
ficiency of DeepTPI. For example, DeepTPI can be paralleled Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski
on multi-devices, which can reduce the runtime. The number et al., “Human-level control through deep reinforcement learning,”
nature, vol. 518, no. 7540, pp. 529–533, 2015.
of GNN iterations can also be reduced with more training [16] M. Li, S. Khan, Z. Shi, N. Wang, Y. Huang, and Q. Xu, “Representation
data. Or we can insert multiple TPs in each inference round. learning of logic circuits,” in 2022 59th ACM/IEEE Design Automation
We leave them for future work. Conference (DAC), to appear, 2022.
[17] J.-S. Yang, N. A. Touba, and B. Nadeau-Dostie, “Test point insertion
VI. C ONCLUSION with control points driven by existing functional flip-flops,” IEEE
Transactions on Computers, vol. 61, no. 10, pp. 1473–1483, 2011.
This paper proposes DeepTPI, a test point insertion ap- [18] Y. Sun, “Novel test point insertion applications in lbist,” 2021.
proach with deep reinforcement learning. As the circuit can [19] J. Fox, “Test-point condensation in the diagnosis of digital circuits,” in
be naturally modeled as a graph and the action is defined as Proceedings of the Institution of Electrical Engineers, vol. 124, no. 2.
IET, 1977, pp. 89–94.
inserting TP on the discrete node over the graph, we train an [20] C. Acero, D. Feltham, F. Hapke, E. Moghaddam, N. Mukherjee,
agent with the combination of deep Q-learning and a graph V. Neerkundar, M. Patyra, J. Rajski, J. Tyszer, and J. Zawada, “Em-
neural network (Graph-DQN) to approximate the value of all bedded deterministic test points for compact cell-aware tests,” in 2015
IEEE International Test Conference (ITC). IEEE, 2015, pp. 1–8.
possible actions. We also embed a testability-aware attention [21] J. Zhang, S. Huynh, and M. Soma, “A test point insertion algorithm for
mechanism into the Graph-DQN to capture the dominant mixed-signal circuits,” in Proceedings 17th IEEE VLSI Test Symposium
impact on testability from both predecessors and successors. (Cat. No. PR00146). IEEE, 1999, pp. 319–324.
[22] C.-Y. Chen and J.-L. Huang, “Reinforcement-learning-based test pro-
The optimal policy is to conduct the action with the maximum gram generation for software-based self-test,” in 2019 IEEE 28th Asian
predicted value. Besides, we apply DeepGate as a pre-trained Test Symposium (ATS). IEEE, 2019, pp. 73–735.
model to learn the universal gate embedding as the prior circuit [23] W. Haaswijk, E. Collins, B. Seguin, M. Soeken, F. Kaplan, S. Süsstrunk,
and G. De Micheli, “Deep learning for logic optimization algorithms,” in
information. The experimental results prove the effectiveness 2018 IEEE International Symposium on Circuits and Systems (ISCAS),
of our strategies and demonstrate that DeepTPI can achieve 2018, pp. 1–4.
better performance than the commercial DFT tool. [24] A. Hosny, S. Hashemi, M. Shalan, and S. Reda, “Drills: Deep reinforce-
ment learning for logic synthesis,” in 2020 25th Asia and South Pacific
R EFERENCES Design Automation Conference (ASP-DAC), 2020, pp. 581–586.
[25] D. S. Lopera, L. Servadei, G. N. Kiprit, S. Hazra, R. Wille, and W. Ecker,
[1] E. J. McCluskey, “Built-in self-test techniques,” IEEE Design & Test of “A survey of graph neural networks for electronic design automation,”
Computers, vol. 2, no. 2, pp. 21–28, 1985. in ACM/IEEE 3rd Workshop on Machine Learning for CAD (MLCAD).
[2] M. He, G. K. Contreras, D. Tran, L. Winemberg, and M. Tehranipoor, IEEE, 2021, pp. 1–6.
“Test-point insertion efficiency analysis for lbist in high-assurance [26] H. Wang, K. Wang, J. Yang, L. Shen, N. Sun, H.-S. Lee, and S. Han,
applications,” IEEE Transactions on Very Large Scale Integration (VLSI) “Gcn-rl circuit designer: Transferable transistor sizing with graph neural
Systems, vol. 25, no. 9, pp. 2602–2615, 2017. networks and reinforcement learning,” in 2020 57th ACM/IEEE Design
[3] M. J. Y. Williams and J. B. Angell, “Enhancing testability of large- Automation Conference (DAC). IEEE, 2020, pp. 1–6.
scale integrated circuits via test points and additional logic,” IEEE [27] A. Mirhoseini, A. Goldie, M. Yazgan, J. Jiang, E. Songhori, S. Wang,
Transactions on Computers, vol. 100, no. 1, pp. 46–60, 1973. Y.-J. Lee, E. Johnson, O. Pathak, S. Bae et al., “Chip placement with
[4] B. Krishnamurthy, “A dynamic programming approach to the test point deep reinforcement learning,” arXiv preprint arXiv:2004.10746, 2020.
insertion problem,” in 24th ACM/IEEE Design Automation Conference.
IEEE, 1987, pp. 695–705.
[28] E. Khalil, H. Dai, Y. Zhang, B. Dilkina, and L. Song, “Learning [36] C. J. Watkins and P. Dayan, “Q-learning,” Machine learning, vol. 8,
combinatorial optimization algorithms over graphs,” Advances in neural no. 3, pp. 279–292, 1992.
information processing systems, vol. 30, 2017. [37] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluation of
[29] V. Kurin, S. Godil, S. Whiteson, and B. Catanzaro, “Can q-learning gated recurrent neural networks on sequence modeling,” arXiv preprint
with graph networks learn a generalizable branching heuristic for a sat arXiv:1412.3555, 2014.
solver?” Advances in Neural Information Processing Systems, vol. 33, [38] W. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation
pp. 9608–9621, 2020. learning on large graphs,” Advances in neural information processing
[30] J. Clifton and E. Laber, “Q-learning: theory and applications,” Annual systems, vol. 30, 2017.
Review of Statistics and Its Application, vol. 7, pp. 279–301, 2020. [39] T. N. Kipf and M. Welling, “Semi-supervised classification with graph
[31] R. Brayton and A. Mishchenko, “Abc: An academic industrial-strength convolutional networks,” arXiv preprint arXiv:1609.02907, 2016.
verification tool,” in International Conference on Computer Aided Ver- [40] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Ben-
ification. Springer, 2010, pp. 24–40. gio, “Graph attention networks,” arXiv preprint arXiv:1710.10903, 2017.
[32] V. Thost and J. Chen, “Directed acyclic graph neural networks,” arXiv [41] S. Devadas, K. Keutzer, S. Malik, and A. Wang, “Certified timing verifi-
preprint arXiv:2101.07965, 2021. cation and the transition delay of a logic circuit,” IEEE Transactions on
[33] M. Zhang, S. Jiang, Z. Cui, R. Garnett, and Y. Chen, “D-vae: A Very Large Scale Integration (VLSI) Systems, vol. 2, no. 3, pp. 333–342,
variational autoencoder for directed acyclic graphs,” Advances in Neural 1994.
Information Processing Systems, vol. 32, 2019. [42] F. Brglez, D. Bryan, and K. Kozminski, “Notes on the iscas’89 bench-
[34] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training mark circuits,” North-Carolina State University, 1989.
of deep bidirectional transformers for language understanding,” arXiv [43] S. Davidson, “Characteristics of the itc’99 benchmark circuits,” in IEEE
preprint arXiv:1810.04805, 2018. International Test Synthesis Workshop (ITSW), 1999.
[35] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, [44] L. Amarú, P.-E. Gaillardon, and G. De Micheli, “The epfl combinational
A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language mod- benchmark suite,” in IWLS, no. CONF, 2015.
els are few-shot learners,” Advances in neural information processing [45] C. Schotten and H. Meyr, “Test point insertion for an area efficient
systems, vol. 33, pp. 1877–1901, 2020. bist,” in Proceedings of 1995 IEEE International Test Conference (ITC).
IEEE, 1995, pp. 515–523.