B LEEM: Packet Sequence Oriented Fuzzing for Protocol Implementations

Zhengxiong Luo1 , Junze Yu1 , Feilong Zuo1 , Jianzhong Liu1 , Yu Jiang1, B , Ting Chen2 ,
Abhik Roychoudhury3 , and Jiaguang Sun1
1 KLISS, BNRist, School of Software, Tsinghua University
2 University of Electronic Science and Technology of China
3 National University of Singapore

Abstract protocol implementations, such as formal analysis [40] and

Protocol implementations are essential components in net- model checking [21]. Though they are somewhat effective
work infrastructures. Flaws hidden in the implementations in some regards, these techniques usually require significant
can easily render devices vulnerable to adversaries. There- prior knowledge to perform.
fore, guaranteeing their correctness is important. However, Fuzzing, an automated software testing technique, has
commonly used vulnerability detection techniques, such as emerged as one of the most effective techniques for detecting
fuzz testing, face increasing challenges in testing these im- vulnerabilities in real-world software. At a high level, given
plementations due to ineffective feedback mechanisms and a target protocol program, a fuzzer works by continuously
insufficient protocol state-space exploration techniques. generating packets and sending them to the program while
This paper presents B LEEM, a packet-sequence-oriented observing for potential anomalies. While traditional protocol
black-box fuzzer for vulnerability detection of protocol im- fuzzers have been widely adopted and detected many vulner-
plementations. Instead of focusing on individual packet gen- abilities, they still suffer from the following problems.
eration, B LEEM generates packets on a sequence level. It pro- First, blind packet generation is ineffective and resource-
vides an effective feedback mechanism by analyzing the sys- consumption-intensive. Traditional fuzzers like Peach [18]
tem output sequence noninvasively, supports guided fuzzing generate packets based on a predefined test model without
by resorting to state-space tracking that encompasses all par- program feedback. Hence, fuzzers are unaware of whether
ties timely, and utilizes interactive traffic information to gen- the generated input triggered new program states. These valu-
erate protocol-logic-aware packet sequences. We evaluate able historical attempts cannot be recognized and utilized for
B LEEM on 15 widely-used implementations of well-known further optimization, lowering their overall efficiency. Some
protocols (e.g., TLS and QUIC). Results show that, compared recent approaches tackle this problem by applying an evolu-
to the state-of-the-art protocol fuzzers such as Peach, B LEEM tionary process: (i) to recognize a successful attempt, they
achieves substantially higher branch coverage (up to 174.93% resort to code coverage [48,58] or status codes extracted from
improvement) within 24 hours. Furthermore, B LEEM exposed server responses [45]; and (ii) to explore further based on
15 security-critical vulnerabilities in prominent protocol im- these successful attempts, they directly retain the inputs in-
plementations, with 10 CVEs assigned. volved and perform variations on them, based on the promise
that these inputs have reproducibility behaviors when passed
to the same target program. However, such a method does
1 Introduction not scale across different protocol implementations: (i) First,
these feedback mechanisms either require instrumenting the
Protocol implementations form the foundations of network
source code or binaries, which may be infeasible when the pro-
infrastructures. Since they are usually exposed directly to
tocol implementation is black-box, or are tightly coupled with
the network, protocol implementations need to handle any
a specific protocol format, which is inappropriate when test-
malformed or malicious traffic from potential attackers cor-
ing a diverse range of protocol implementations. (ii) Second,
rectly. Any flaws undetected in the implementations (0-days)
some protocols’ specific validation rules can render existing
can easily render devices vulnerable to adversaries. For in-
evolutionary procedures futile. For example, some common
stance, the infamous Heartbleed [15] vulnerability discovered
protocols, such as TLS [3], DTLS [4], and SSH [39], employ
in OpenSSL can cause sensitive data exposure. Many tech-
random nonces during the handshake to prevent “replay at-
niques have been proposed to increase the security of these
tacks”. In this scenario, the valuable inputs retained in the pre-
B Yu Jiang is the correspondence author. vious exploration can no longer replay the interesting behav-
iors it once triggered, making the input generation attempts cies and avoiding producing meaningless packets.
based on these inputs less likely to produce valuable results. We evaluated the performance of B LEEM on 15 widely-
Second, since protocol implementations are stateful sys- used implementations of well-known protocols, including
tems, the input space of the under-test program is tightly QUIC, TLS, and DNS. We compared B LEEM against five
regulated by its state. Effectively traversing the massive state-of-the-art protocol fuzzers, including Peach [18], Boo-
state space and covering the versatile communication reg- Fuzz [33], AFLNet [45], SGFuzz [8], and Snipuzz [20]. The
ulations (i.e., various transitions) requires carefully-crafted experimental results demonstrate that B LEEM outperforms
packet sequences. However, constructing such packet se- prior works in branch coverage by 28.5%, 48.9%, 35.7%,
quences is non-trivial because it involves complex protocol 23.4%, and 40.3% on average, respectively, over 24 hours.
logic, e.g., in what order to transmit the packets and how to We also show that the proposed fuzzing strategy can effec-
construct them to guarantee the format and parameter correct- tively help B LEEM explore state space. B LEEM exposed 15
ness. Some existing approaches, such as Peach [18], resort to new vulnerabilities in previously well-tested real-world proto-
a user-defined protocol model and generate packet sequences col implementations, and Peach, BooFuzz, AFLNet, SGFuzz,
by strictly following the actions depicted in that model. While and Snipuzz can only expose 8, 5, 6, 7, and 5 of them, re-
the protocol logic involved in the model can be covered ef- spectively. Most of these vulnerabilities are security-critical,
fectively, logic beyond the confines of the model cannot be where 10 CVEs were assigned due to their severe security
exercised. Other approaches, such as AFLNet [45], generate influences. In summary, our main contributions are as follows:
packet sequences by performing variants on existing packets. • We develop a versatile technique to collect feedback by
Due to unawareness of the protocol format, these approaches analyzing the system’s output sequence.
remain limited in providing valid inputs for protocol imple- • We design the SSTG that identifies the observed state space
mentations that process highly-structured packets. to navigate the fuzzing exploration direction and propose a
In this paper, we present B LEEM, a packet-sequence- protocol-logic-aware packet sequence generation method
oriented black-box fuzzer, to address the problems above. based on real-world prior information.
B LEEM employs an on-the-fly noninvasive feedback collec- • We implement B LEEM and evaluate it on widely-used
tion mechanism and conducts dynamic state-space tracking protocol implementations. The results demonstrate that
of all parties (i.e., both the client and server) from the system- B LEEM outperforms the state-of-the-art and has detected
under-test to guide the packet sequence generation. many security-critical vulnerabilities.
First, we introduce a noninvasive on-the-fly feedback mech-
anism by analyzing the system’s output. The insight is that the
outputs of protocol implementations (i.e., packet sequences) 2 Protocol Fuzzing
can abstract the inner protocol state while also being typically
structured. Hence, we collect the output sequence of the tar- Traditional protocol fuzzers focus mainly on testing server-
get system at runtime, analyze the semantics conveyed in the side implementations. They act as clients, constantly generat-
output, and leverage it to indicate the internal system state ing packets and sending them to the servers. Based on how the
transitions from the perspective of considering the protocol packets are produced, these fuzzers can be roughly classified
parties as a whole system. into two categories: mutation-based and generation-based.
Second, we leverage heuristics to guide the packet sequence
generation based on the feedback. We devise the System State 2.1 Mutation-based Fuzzers
Tracking Graph (SSTG), which is dynamically constructed at
runtime and allows B LEEM to chart the explored state space. These fuzzers generate new inputs by randomly mutating
Navigated by the SSTG, we facilitate a comprehensive traver- existing inputs selected from a corpus [20, 37, 43, 45, 48, 58].
sal to increase the exposure of different protocol behaviors They require no prior knowledge of protocol specifications
and explore the unknown state space by providing diverse in- and message formats, thus, are easy to deploy. These fuzzers
puts for different states. To this end, we extend the SSTG with are good at testing stateless programs (e.g., file processing
richer information to enable it to provide guidance for reach- application) where no internal state is maintained. To fuzz
ing desired states and introduce mutation operators working the protocol implementations, developers use a workaround,
on different levels, including the packet and sequence levels, where writing test harnesses that implement the unit testing
to support customized packet generation on different states. of specific server state or simply treating the input sent to the
Meanwhile, we leverage information extracted from the inter- server as a concatenation of messages to implement system
active traffic of the protocol to generate packet sequences. We testing. For better performance, AFLNet [45] borrows the
observe that since the protocol logic is implemented within mainstream design that augments the traditional fuzzers with a
the server and client, we can generate protocol-logic-aware feedback loop [58], enabling the fuzzers to track the execution
packet sequences based on the observed traffic exchanged information exercised by each input and retain the inputs that
between the two parties, thus preserving parameter dependen- contribute to new program behaviors for further utilization.
Also, it integrates state awareness by analyzing the status Initial[CRYPTO] packet (Line 4); then, wait for responses
codes in server responses. Nevertheless, lacking the format from the server and check whether the responses conform to
specification, these fuzzers can quickly encounter barriers the format of the given data model Initial[CRYPTO,ACK]
because the packets generated from blind mutation can be and Handshake[CRYPTO] (Line 7); if so, enter the third Ac-
easily dropped by the protocol program that processes highly- tion and send Initial[ACK] and Handshake[CRYPTO,ACK]
structured packets. Furthermore, these feedback mechanisms packets (Line 10, the data models of other kinds of packets,
do not apply to black-box fuzzing, and their evolutionary e.g., Initial[CRYPTO,ACK], have been omitted in Figure 2
methods can be futile by the special checks (discussed in §1). due to limited space).
1 < StateModel name =" QUIC " initialState =" HANDSHAKE " >
2 < State name =" HANDSHAKE " >
2.2 Generation-based Fuzzers. 3
4
< Action type =" output " >
< DataModel ref =" Initial [ CRYPTO ]"/ >
5 </ Action >
These fuzzers generate the packets based on manually con- 6 < Action type =" input " >
7 < DataModel ref =" Initial [ CRYPTO , ACK ]+ Handshake [ CRYPTO ]"/ >
structed protocol models, including data and state mod- 8 </ Action >
9 < Action type =" output " >
els [6, 18, 33, 53]. The state model is typically given in the 10 < DataModel ref =" Initial [ ACK ]+ Handshake [ CRYPTO , ACK ]"/ >
11 </ Action >
form of a graph and specifies the valid message sequencing 12 </ State >
in the interaction with the server. The data model depicts the 13 ...
14 </ StateModel >
formats (e.g., field types, field sizes, and valid value ranges) 15
16 < DataModel name =" Initial [ CRYPTO ]" >
of accepted messages for the corresponding states. The user 17 < Block name =" Header " >
needs to know the protocol interaction logic to provide this 18 < Number name =" info " size ="8" value =" cd " valueType =" hex "/ >
19 < Number name =" version " size =" 32 " value =" faceb002 "
test model by analyzing the source code or reading the proto- valueType =" hex " token =" true "/ >
20 < Number name =" DCID_length " size ="8" >
col specification. In the following, we take the QUIC hand- 21 < Relation type =" size " of =" DCID "/ >
22 </ Number >
shake as an example and give the corresponding test model 23 < String name =" DCID " nullTerminated =" false "/ >
for Peach [18] initialization. 24 ...
25 </ Block >
26 < Block name =" CRYPTO " > ... </ Block >
Client Server 27 </ DataModel >
Initial[CRYPTO]
①
Initial[CRYPTO, ACK], Figure 2: A Simplified QUIC Handshake Model as Peach Pit.
Handshake[CRYPTO]
②
Initial[ACK], The generation-based fuzzers have successfully exposed
Handshake[CRYPTO, ACK]
③ plenty of vulnerabilities in real-world protocol implementa-
1-RTT[APPLICATION_CLOSE]
④ tions. However, their effectiveness is heavily limited by the
quality of the user-provided protocol model, whose construc-
Figure 1: An example dialog between the QUIC server and client tion usually requires significant expert efforts to read the
protocol specification or the source code. First, such specifica-
Figure 1 depicts a sample dialog between a client- tions are prone to misinterpretation because they are typically
server pair compliant with QUIC specification (as per the expressed in a natural language, thus coming with inherent
RFC 9000 [32]). The QUIC client starts by sending an ambiguities. Second, the protocol generally features a massive
Initial[CRYPTO] packet ( 1 , Initial[CRYPTO] denotes state space, making it infeasible to construct a complete model
that the packet is of type Initial and contains a CRYPTO that describes all the protocol behaviors. For example, Fig-
frame, which represents the packet on an abstract level by ure 2 only depicts the handshake logic of QUIC and does not
omitting some detailed fields). Then the server responds with cover many other logics, such as flow control. Unfortunately,
Initial[CRYPTO,ACK] and Handshake[CRYPTO] packets, these fuzzers conduct testing by strictly following the actions
carrying the essential information for handshake estab- depicted in the protocol model (see Figure 2, Lines 1-14),
lishment ( 2 ). The client then sends Initial[ACK] and and they do not update the state model at runtime. Therefore,
Handshake[CRYPTO,ACK] packets, indicating the comple- the new interesting program behaviors beyond those intro-
tion of the handshake ( 3 ). For simplification, we as- duced by the deterministic model cannot be recognized by
sume the server is set to terminate the connection via these fuzzers and therefore cannot be utilized for further ex-
1-RTT[APPLICATION_CLOSE] after handshake ( 4 ). ploration. For example, say Peach, in the configuration of the
Figure 2 depicts the QUIC handshake model ( 1 - 3 in Fig- Pit in Figure 2, accidentally triggers a new program behav-
ure 1) in the format of Peach Pit [54], the XML configuration ior via some interesting Initial[CRYPTO] packet, and the
file for the protocol fuzzer Peach [18]. Lines 16-27 show server responds with another packet not defined in the state
the data model of packet Initial[CRYPTO], which is sent at model instead of the desired Initial[CRYPTO,ACK] packet.
the first Action (Lines 3-5) of the State HANDSHAKE (Lines 2- Peach then discards it and continues to perform the determin-
12). Specifically, this State specifies how Peach interacts with istic testing behaviors instead of exploring more interesting
the QUIC server to test the handshake phase: first, send the behaviors based on this discovery. Third, some illegal be-
haviors in the protocol implementation may be introduced logic, i.e., in what order packets are transmitted for differ-
by implementation errors. These behaviors violate the stan- ent server states and how these packets are built to guaran-
dards and thus are not included in the protocol specification. tee parameter correctness. Instead of manually implementing
Hence, these scenarios cannot be covered under the model such logic from scratch, as is the case with existing black-
constructed based on the protocol specification. box techniques (e.g., Peach Pit in Figure 2), we argue that
the client already encodes the desired protocol logic and can
2.3 Challenges therefore be utilized to provide information for packet gen-
eration. Specifically, B LEEM requires a pair of client and
In order to design an effective and efficient fuzzing method server that can communicate with each other and considers
that can scale across diverse protocol implementations, we this pair as a whole system to analyze. Meanwhile, unlike
need to overcome two challenges. existing coverage-guided works requiring access to the source
1) Lack of a scalable feedback mechanism. Although the code [45] or the binary [48] of the implementation, B LEEM
feedback-driven approach has shown effectiveness in opti- can be conducted in a black-box manner. All we assume is
mizing fuzzing [58], the versatile applications of protocol that B LEEM can interact with the parties of SUT.
implementations stress the scalability of the traditional feed-
Bleem Overview
back mechanisms. Nowadays, protocol implementations have
been widely applied in industrial programmable logic con- Feedback Collector Guided Fuzzing

troller (PLC) devices and Internet of Things (IoT) devices. State Trace Transition
Test Priority
Instrumenting the device’s firmware is challenging and even Initilization
infeasible when the implementation is only accessible in a System State Mutation
Abstract Packet
Tracking Graph Operators
Sequence
black-box fashion, which necessitates a noninvasive solution
to obtain feedback for the protocol fuzzers. Analyze 1
Bug Oracle Packet Pattern
2) Highly complex protocol logic. Protocols feature highly Report
Map Sequence

complex logic designed to guarantee correctness and relia-

Network
bility under diverse situations, making the implementations Traffic Packet Instantiation
highly complex and stateful interactive targets. Therefore,
in-depth implementation fuzzing requires carefully construct-
ing the packet sequence by considering the protocol logic, Client System Under Test (SUT) Server
including the packet format, the parameter dependency be-
tween packets, and the packet interaction logic. Specifically, Figure 3: B LEEM Overview. The Feedback Collector monitors
the packet format identifies the standard packet structure, the the SUT for feedback extraction, based on which the Guided
packet interaction logic identifies the transmission order of dif- Fuzzing module tracks the state space and applies heuristics
ferent types of packets, and the parameter dependency speci- to generate packet sequences by leveraging protocol-aware
fies the value dependencies of parameters exchanged between mutation operators and the interactive traffic with the SUT.
packets. For example, in Figure 1, the Initial[CRYPTO] In most cases, providing such a client-server pair requires
packet at 1 and the Initial[ACK] packet at 2 should carry low effort. Protocol implementations usually provide users
the same value for the parameter DCID, which is a connec- with client and server utilities, which can be used directly
tion identification field. Otherwise, the handshake would fail, to compose the SUT. When one side of the pair is missing,
making the following test actions meaningless. we can resort to any other valid implementation of the same
protocol since the protocols are generally standardized. In
3 System Overview the worst case, when off-the-shelf utilities are unavailable on
either side, the user needs to provide such a pair since a test
Figure 3 illustrates the overview of B LEEM: given the Sys- entry is a prerequisite for fuzzing, in line with other existing
tem Under Test (SUT) of the target protocol implementation, fuzzers [9, 31, 34, 59, 60].
B LEEM is initialized with the Test Initialization and outputs Test Initialization. Unlike existing black-box techniques
the Bug Report via packet-sequence-oriented fuzzing. that require manually constructed protocol models, B LEEM
System Under Test (SUT). The SUT is the protocol im- starts fuzzing based on the dialog captured by executing the
plementation we intend to analyze. Such protocols usually SUT, thus is easier to set up. Since testing the protocol im-
adhere to the client/server model. Given a protocol implemen- plementation by directly launching the SUT is improbable to
tation, traditional fuzzers mainly target one protocol side and reveal flaws, as such compliant cases should have been cov-
perform tests by acting as another. For example, to test the ered in pre-release tests, we need to generate different variants
server side, the traditional protocol fuzzer acts as a client and as inputs for the SUT parties. To this end, instead of letting the
continuously sends generated packets. In this scenario, effec- two parties communicate normally, B LEEM interacts directly
tive server fuzzing requires expert knowledge of the protocol with both the server and the client, which enables intercept-
ing their exchanged traffic and introducing adversely crafted Handshake[CRYPTO,ACK] packets, we can infer that it has
packets for testing. Therefore, for test initialization, we need completed the handshake and is ready to enter the data trans-
to provide B LEEM with the server’s service address (e.g., the mission phase. Furthermore, the packets’ structure is typically
socket address) and configure B LEEM with a different service standardized and protocol-compliant, thus facilitating analy-
address for the incoming connection from the client. In this sis. Based on this insight, B LEEM works as shown in Figure 4.
way, B LEEM can collect the output packets sent by the SUT #
𝑃!" !
𝑃'" !
𝑃!" !
𝑃!$ #
𝑃'" #
𝑃'$ #
𝑃!" !
𝑃!" !
𝑃!$ State Trace
❶Filter
parties. These packets are then utilized for SUT inner state ❷Abstract ❹Construct
analysis and serve as a basis for packet generation. B LEEM BLEEM Fuzzing #
𝑃⍵" !
𝑃&" !
𝑃&$ !
⍵" ⍵$ (𝑃&" !
+𝑃&$ )
❸Process
leverages Scapy’s [50] parsing capability to better analyze the
intercepted packets. For unsupported protocols, we can extend Figure 4: Feedback collector workflow
Scapy by identifying the protocol format (i.e., data model), Filtering the Packet Sequence. In Figure 4, Ps1 c denotes
which is more accessible than traditional generation-based s
the 1st packet sent by the client, while Pr1 donates the 1st
fuzzers that require both the data model and the state model. packet received by the server. Given the network traffic cap-
Meanwhile, we can simplify this step by resorting to protocol tured during fuzzing, the feedback collector picks out the
reverse engineering, as discussed in §8. packets associated with the SUT state to assemble a packet
Workflow. As shown in Figure 3, B LEEM consists of two sequence S , where the relative order of these packets remains
components: the feedback collector and the guided fuzzing the same as in the original traffic (¶). In detail, we only
module. In each iteration, the noninvasive feedback collec- consider the packets sent by the SUT parties (e.g., Ps1 c ) and
tor captures the network traffic during the SUT execution c
discard their received packets (e.g., Pr1 ), which B LEEM sends
until it finishes. After that, it analyzes the traffic, extracts mes- for fuzzing purposes and thus cannot reflect the internal state.
sage semantics to construct an abstract packet sequence, and Abstracting the Packet Sequence and Processing. To
translates the abstract packet sequence to a state trace, which indicate the SUT states, using the concrete packets directly
incorporates both the client and server and serves as the SUT can create confusion because some fields, e.g., the data field,
feedback to be passed to the guided fuzzing module. own low association with the system state, and their possible
The guided fuzzing module then merges the state trace to values are sometimes infinite. Take the Initial[CRYPTO]
a System State Tracking Graph (SSTG), refines the SSTG’s packet in Figure 2 as an example. The value of the DCID field
traverse probability distribution, and applies a guided fuzzing used to identify a connection (Line 23) is randomly generated
strategy to select mutation operators and generate packet pat- at the start of the connection [32]. Considering this field can
tern sequence to be instantiated. Specifically, we devise the make the state space too big or infinite. Therefore, we need to
SSTG to describe the all-inclusive state-space of the SUT. abstract away some details and focus on the crucial semantic
B LEEM initializes the SSTG with the initial session embed- information carried by the packets.
ded in the SUT and dynamically updates it with new findings, We abstract the packets in S one by one to obtain an el-
i.e., new states or transitions, discovered in the feedback. Then ementary abstract packet sequence π (·). For each packet
B LEEM instantiates the packet pattern sequence based on the P ∈ S , the feedback collector employs Scapy to parse it and
packets exchanged between the SUT parties. It simultane- gets the dissection Pd . The dissection result is a hierarchi-
ously interacts with both the client and server to intercept cally organized list of fields, where each field is represented
the exchanged packets, which are typically syntactically and as a type-value pair. Based on Pd , the module constructs the
semantically correct. Thus, the packet sequences generated corresponding packet abstraction Pω by retaining these fields
based on them have a high probability of exercising deep pro- of type enumeration, which is a general field type defined in
tocol logic. We dive into the details in the following sections. Scapy to describe the fields whose valid values are taken from
a fixed small set. Through our investigation of over 50 sup-
4 Feedback Collector ported protocols in Scapy, we found that the different values
of the enumeration field typically represent different types of
To augment B LEEM’s capability of obtaining feedback when packets or frames (some protocols’ packet payload consists
fuzzing the targets of black-box nature, we propose to re- of a sequence of complete frames, like QUIC). In this way,
trieve system feedback based on system output instead of B LEEM abstracts the first flight of the QUIC handshake ( 1 in
invasive execution inspection. Unlike ordinary application Figure 1, Lines 16-27 in Figure 2) as Initial[CRYPTO] by
programs, the protocol implementations are tailored for entity retaining the packet type Initial and frame type CRYPTO and
communication over networks. Hence, the program output, their hierarchical relationships. We also provide a detailed ex-
i.e., packets, can be adopted as the indicator of its inner state. ample in Figure 11 in Appendix D. Moreover, B LEEM caches
Taking the QUIC session in Figure 1 as an example, when an intermediate map between each abstract packet Pω and
the client sends an Initial[CRYPTO] packet to the server, its corresponding recent concrete packet P in a data structure
we can infer that the client may stay in the initial state of called Oracle Map, which is further explored in §5.4.
the connection. When the client sends Initial[ACK] and Then, to facilitate the state trace construction, we further
process the abstract packet sequence π by concatenating these 5 Guided Fuzzing
adjacent abstract packets Pω s sent from the same source Empowered by the feedback collector, B LEEM can track the
into one abstract packet ω (¸). For example, in Figure 1, state transitions of the SUT and determine whether the SUT
the two packets at 2 can be abstracted and concatenated reaches a new, previously untraversed state region. We de-
into Initial[CRYPTO,ACK]+Handshake[CRYPTO]. There- vise the SSTG to represent the explored all-inclusive system
fore, in the final abstract packet sequence π, the adjacent state-space formally. Taking the perspective of the SSTG, we
two abstract packets are sent by different protocol parties. introduce protocol-aware mutation operators to support di-
Constructing the State Trace. For a given abstract packet verse packet generation for different states. We also extend
sequence π : {ω1 , ω2 , . . . , ωn }, each abstract packet ωi can the SSTG with richer information to embrace the ability to
only indicate the temporal state of the corresponding SUT provide a guideline for reaching the desired state. We design a
party pi instead of the whole SUT. Because there may be guided sequence generation strategy to effectively explore the
multiple paths that can reach this temporal state for the party state space and leverage the packet instantiation sub-module
pi , but their corresponding temporal SUT states at the point ωi to provide high-quality packet sequences.
vary. For example, assume that the client sends either packet
a or packet b (a and b are different types) to the server at a
certain point, and both packets are unexpected to the server.
5.1 Mutation Operators
In this case, the server would respond to either with the same We introduce protocol-aware mutation operators that work on
error message c. When the server sends out packet c, the different levels, including the packet and sequence levels.
whole SUT state should be different under these two different Packet-Level Mutation Operator. This mutation opera-
paths. Hence, we treat the protocol fuzzing as a bipartite SUT tor (denoted as σP ) operates on the fields in the packet. For
and model its state by introducing some prior information. a given packet, it randomly selects several fields and per-
However, considering a complete history can cause state space forms corresponding mutation operations according to the
explosion. We have found that considering the most item field type. Scapy has identified five general field types, includ-
in the bi-directional communication is enough to strike a ing NumberField, StringField, ListField, EnumerationField,
practical balance in indicating the temporal state of the whole and LengthField. Thus, based on their features, we devise
SUT while minimizing the overall state tracking space. mutation operators for them. For example, the NumberField
mutation operator performs random addition or subtraction
Definition 1 SUT State. A SUT State incorporates both par-
operations to the original value while considering the valid
ties of the SUT and can be represented as an ordered pair
value range. We provide further details in Appendix A.1.
of objects in the form of hOb j1 | Ob j2 i. Each object is taken
Sequence-Level Mutation Operator. This mutation oper-
from the set {T (ω) | T ∈ {C, S}, ω ∈ Ω}, where C represents
ator (denoted as σS ) operates on the packets in the sequence.
the client while S represents the server, and Ω is an abstract
B LEEM offers the following two kinds of operators:
packet alphabet. Meanwhile, the order of these two objects is
1. Packet duplication. Given a packet sequence, this opera-
significant: hT1 (ω1 ) | T2 (ω2 )i represents that the party T1 is
tor selects an arbitrary packet Pi and, after it, introduces
going to respond ω1 (the prior), with the precondition that the
several duplicate copies of Pi (denoted as Pi∗ ). Figure
party T2 has sent ω2 (the posterior). Note that ω2 only records
5(a) gives an example. The original sequence is [P1 , P2 ,
the most recent abstract packet delivered by T2 .
P3 , P4 . . . ]. If selecting P2 and duplicating it twice, the
In this way, we can infer the executed SUT States by analyz- sequence after mutation would be [P1 , P2 , P2∗ , P2∗ , F5 . . . ].
ing each pair of adjacent abstract packets ωi and ωi+1 (their Note that the subsequent packets in the sequence may
sources are opposite as guaranteed above) in π. Then, adding change after duplication, and we introduce the symbol “F”
transitions between two adjacent SUT States forms a state to represent the packets lying after the manipulated pack-
trace (¹). We represent the involved SUT states in Figure 1 ets. In this case, the packet following the intentionally
using an abstract packet alphabet Ω with 5 symbols: introduced P2∗ s may no longer be P3 in real traffic. We use
Ω = {∅ : NoPacket, a : Initial[CRYPTO], F5 to represent the fifth packet in the mutated sequence.
b: Initial[CRYPTO,ACK]+Handshake[CRYPTO], 2. Packet disordering. Given a packet sequence, this operator
(1) selects an arbitrary packet Pi (the i-th one in the original
c: Initial[ACK]+Handshake[CRYPTO,ACK],
order) and changes its order to be delivered. Similarly,
d: 1-RTT[APPLICATION_CLOSE]},
Figure 5(b) gives an example, the fourth packet P4 in the
The SUT State at point 1 can be represented as hC(a) | S(∅)i,
sequence [P1 , P2 , P3 , P4 . . . ] is selected to be delivered after
indicating that the server has sent no packet while the client is
P1 , namely deleting P2 and P3 and yielding the mutated
going to respond a (Initial[CRYPTO]). Similarly, the SUT
sequence [P1 , P4 , F3 , F4 . . . ].
State at point 2 can be represented as hS(b) | C(a)i. In this
These two sequence-level mutation operators are designed
way, the state trace constructed for this session would be:
to stress test anomalies under uncertain network environments
hC(a) | S(∅)i → hS(b) | C(a)i → hC(c) | S(b)i → hS(d) | C(c)i (2) that may introduce dynamic packet delays or losses, espe-
cially for the protocols over UDP. Specifically, the packet obtained from the feedback collector, B LEEM labels the trig-
disordering operator enables the discovery of anomalies trig- gering condition of each transition. Instead of retaining the
gered by the abnormal order of packets carrying commands. concrete packets directly, as existing approaches do, B LEEM
For instance, the File Transfer Protocol (FTP) applies a set records how to generate these packets using the correspond-
of commands to comprise the control information flowing ing abstract packet ω and the mutation operator σ selected to
from the user-FTP to the server-FTP process [47]. Con- operate on it, i.e., packet pattern ω ⊕ σ. The abstract packet
sider a simplified legal command sequence from the user- ω can be extracted from the source SUT State as B LEEM
FTP: [USER test, PASS test, STOU test.txt, . . . , DELE generates packets based on the packet received from the SUT
test.txt], which means that the user-FTP first logins to the parties. The corresponding mutation operator σ can be ob-
server-FTP with username test and password test, then tained from B LEEM’s execution record. Meanwhile, we do
uploads a file test.txt, performs some actions, and finally not track the details of the mutation operators to avoid the
deletes it. Due to possible malicious actions induced by adver- state space explosion problem.
saries, the server-FTP implementation should correctly handle q0 q1 q2 q3
a⨁𝜎∘ b⨁𝜎∘ c⨁𝜎∘
bad command sequences, e.g., deleting the PASS test in the C(a) | S(Ø) S(b) | C(a) C(c) | S(b) S(d) | C(c)

sequence to attempt an authentication bypass. The packet

sequence given in this example is one-way for ease of under- Figure 6: The initial SSTG for the QUIC session in Figure 1
standing. In practice, B LEEM operates over a two-way packet For example, we label the state trace in Formula 2, which is
sequence, i.e., packets exchanged by the two parties. equivalent to B LEEM directly forwarding the received packets
Original
P1 P2 P3 P4 … P1 P2 P3 P4 …
without mutation (i.e., σ◦ ). Figure 6 shows the result. In detail,
sequence
the SUT starts at the state q0 , where the client is going to
output packet a. After receiving a, B LEEM generates packets
Mutated as input for the SUT by mutating a using σ◦ , namely a ⊕
P1 P2 P2* P2* F5 … P1 P4 F3 F4 …
sequence
σ◦ (actually, a⊕σ◦ is delivered to the server at the micro level,
(a) Packet duplication (b) Packet disordering
which is achieved by the packet instantiation sub-module in
Figure 5: Sequence-level mutation operators §5.4). After reading the input a ⊕ σ◦ , the SUT transitions to
the state q1 . The remaining states are traversed similarly.
Formalization. To facilitate the representation, we can also Merging the State Trace. Then, we use the SSTG to track
regard the sequence-level mutation operator as operating on the explored state space by continually merging the state trace.
the involved packet, in line with the packet-level mutation op- Definition 2 System State Tracking Graph (SSTG). A SSTG
erator. For example, the packet duplication in Figure 5(a) can is a variant of nondeterministic finite automaton (NFA), and
be considered as operating on P2 . Similarly, the packet disor- can be represented formally by a 5-tuple (Q, q0 , Ω, Σ, ∆), such
dering shown in Figure 5(b) can be considered as operating that: Q is a finite set of the SUT State, q0 is the initial (or start)
on P2 , i.e., substituting P2 with cached P4 in the Oracle Map. state, Ω is an abstract packet alphabet, Σ is a set of mutation
Based on this premise, we introduce a symbol ⊕ to associate operators, and ∆ defines a transition function: Q × {Ω ⊕ Σ} →
the packet P with the adopted mutation operator σ. In this P (Q), where P (Q) denotes the power set of Q. Due to the
way, P ⊕ σ can be used to represent the packet(s) generated inherent randomness of any mutation operator σ ∈ {Σ \ σ◦ },
by mutating P using σ (we use “packet(s)” because the packet for an abstract packet ω ∈ Ω, the set of packet(s) that can be
duplication mutation operator can generate multiple packets). generated using packet pattern ω ⊕ σ is infinite. Hence, the
Hence, P ⊕ σ can be considered a packet pattern, which can SSTG is nondeterministic.
be employed to create new packet(s). Additionally, we intro- B LEEM initializes the SSTG using the state trace defined
duce σ◦ to indicate performing no mutation operation. Due to in the initial dialog. For example, if B LEEM conducts fuzzing
the inherent randomness of any mutation operator σ (σ 6= σ◦ ), based on the dialog in Figure 1, Figure 6 gives the initial
the set of the packet(s) that can be generated by the packet SSTG over the alphabet Σ = {σ◦ } and Ω given in Formula 1.
pattern is typically infinite. As a further generalization, the B LEEM interacts with the SUT parties during the fuzzing
involved packet P can also be given on an abstract level. In and generates diverse packets for the SUT parties instead of
this case, it is necessary to instantiate P before applying the forwarding the received packets directly (which is equiv-
mutation operators. alent to establishing the initial dialog). Although this may
bring on different state traces, these state traces have the same
5.2 System State Tracking Graph start state. Because B LEEM conducts fuzzing based on the
interactive traffic, the launch of each iteration is driven by
We devise the SSTG to identify the explored state space of the first action of the SUT. For example, as for the fuzzing
the SUT and extend it with richer information to mitigate the based on the dialog in Figure 1, the client would always send
reproducibility problem mentioned above. the Initial[CRYPTO] packet to B LEEM at the beginning of
Labeling the State Transitions. First, for the state trace each iteration. The difference lies in the follow-up actions
of B LEEM, e.g., how B LEEM mutates the received packets. applied (Line 1). If so, we construct a packet pattern sequence
Therefore, in order to merge the state trace into the imple- ϕ that can reach q and then append the desired packet pattern
mented SSTG, B LEEM starts from SSTG’s start state, which after ϕ (Line 4). For example, for the SSTG in Figure 6 with
is also the start state of the state trace to be merged, merges Σ0 = {σ◦ , σP , σS }, if stressing q1 with a new packet pattern
their shared nodes or transitions, and updates the SSTG with that combines q1 ’s output b with an unused mutation operator
new states or state transitions discovered in the state trace. σP , i.e., b ⊕ σP , then we can construct such a packet pattern
Furthermore, B LEEM implements a power schedule that sequence: [a ⊕ σ◦ , b ⊕ σP ]. Second, after all the SUT states
dynamically refines the priority of exercised transitions during have been exercised by all kinds of packet patterns that are
the merging. For a transition τ with exercised times Tτ , its available, we attempt to facilitate comprehensive traversal
priority is calculated by h(Tτ ), where h(x) is a decreasing of the implemented SSTG by steering towards low-density
function. That is to say, the newly added transition during regions. We start from the initial state (Line 7) and then run
merging will be assigned the highest priority. in a loop until reaching an end state (Line 8): in each step,
The SSTG captures the exercised state space of the protocol we select the one with the highest priority among the avail-
implementation and provides an efficient way for the state able transitions of the corresponding state (Line 9), record
space traversing: given a target state qi to be reached, the the corresponding packet pattern labeled on it (Line 10), and
transition labels on any trace from q0 to qi form a packet take this transition (Line 11). It is worth noting that even if
pattern sequence, which provides a guideline for the SUT we have once applied a packet pattern P to stress a state S
input generation. and failed to discover new behaviors, it is also worthwhile
trying P on S later because, as mentioned above, the set of
5.3 Guided Sequence Generation the packet(s) that P can produce is typically infinite.
Navigated by the SSTG, we devise heuristics to efficiently
explore unknown state space and facilitate comprehensive 5.4 Packet Instantiation
traversal. This module outputs the packet pattern sequence
and passes it to the packet instantiation sub-module as a guide- For a given packet pattern sequence ϕ : [ω1 ⊕ σ1 , ω2 ⊕
line for packet sequence generation. σ2 , . . . , ωn ⊕ σn ], the packet instantiation sub-module gener-
ates a concrete packet sequence conforming to ϕ as the SUT
Algorithm 1: Guided packet sequence generation input. Since the packets ωi s in ϕ are given on the abstract
Input: SSTG: System State Tracking Graph level, we need to instantiate them while maximally guarantee-
Input: PT : transition priority ing their syntactic and semantic correctness. To this end, this
Output: ϕ: packet pattern sequence module utilizes the protocol logic encoded in the SUT parties.
1 S ← T RAVERSE (SSTG) It runs as a proxy and simultaneously interacts with both the
2 if S is not empty then client and server, thereby leveraging their exchanged packets
3 q ← S ELECT O NE(S ) and introducing crafted packets.
4 ϕ ← E NTER A NDA PPEND(SSTG, q) The adjacent packet patterns in ϕ are prepared for different
5 else SUT parties, as guaranteed by the SSTG. The packet instanti-
6 ϕ←∅ ation sub-module instantiates them one by one. For example,
7 q ← G ET I NITIAL S TATE(SSTG) for a packet pattern ωi ⊕ σi to be instantiated as server input,
8 while H AS S UCCESSOR(q) do the packet instantiation sub-module intercepts a packet P (if
9 τ ← S ELECT T RANSITION(SSTG, PT , q) there are multiple packets, concatenate them into one) from
10 ϕ ← ϕ ∪ G ET PACKET PATTERN(τ) the client and checks whether P conforms to ωi ’s structure:
11 q ← T RANSITION(SSTG, q, τ) (i) if so, it directly takes P as the instantiation of ωi and then
performs the corresponding mutation operator σi on P to gen-
12 return ϕ
erate test packet; (ii) if not, indicating that the realistic inner
Algorithm 1 provides an overview of the process: first, we state transition of the SUT differs from the one designed by
attempt to stress test each SSTG state with diverse inputs by ϕ, there are two possible situations: (a) some new states or
utilizing the proposed mutation operators (Lines 1-4). Specif- state transitions are discovered; and (b) due to SSTG’s in-
ically, for a SUT State hC(α)|S(β)i, although the state output herent non-deterministic, some transitions were not taken as
α is “deterministic” (the corresponding concrete packets of expected, causing a different state trace. In these situations, to
each execution may vary in some details) such that B LEEM strike a balance between comprehensive traversal of the im-
receives the same packet type α as the mutation basis, we plemented SSTG and efficient exploration of unknown state
can select a different mutation operator σ to construct a dif- space, B LEEM randomly selects the following two strategies
ferent packet pattern α ⊕ σ as test inputs. To this end, we to generate the concrete packets: (1) neglect ωi and perform
first check whether there is a state q that can be further ex- σi on P directly; and (2) resort to the Oracle Map for the
ercised by other kinds of packet patterns that have not been packet conforming to ωi and perform σi on it.
6 Evaluation Crash Detection. Besides the wildly-used local process
monitoring, we also implement several network monitors
In this section, we implement and evaluate B LEEM to answer that can remotely detect system crashes to support black-box
the following three research questions: fuzzing. We provide the details in Appendix A.3.

RQ1 Is B LEEM more effective and efficient than traditional

protocol fuzzers? (§6.3) 6.2 Experiment Setup
RQ2 Is B LEEM effective in exposing unknown vulnerabili- Subjects. We measured the performance of B LEEM on two
ties in real-world protocol implementations? (§6.4) benchmarks. (i) Open-source protocol implementations. We
RQ3 Does the guided packet sequence generation strategy selected several open-source implementations of widely used
contribute to the effectiveness of B LEEM? (§6.5) protocols, as shown in Figure 7 and Table 5 in Appendix B.
The selected protocols are pretty diverse, ranging from secu-
rity protocols to messaging protocols, transport protocols, and
6.1 Implementation
industrial control system protocols. These protocols’ corre-
We implement a prototype of B LEEM in Python 3 and use a sponding implementations are typical and widely used in prac-
modular approach to facilitate further extensions. tice. To facilitate a fair comparison, all the selected SUTs (cf.
Feedback Collector. We implement the analysis of col- the fourth column of Table 5) come from off-the-shelf util-
lected packets based on the packet parsing capability provided ities. (ii) Closed-source protocol implementations. We also
by Scapy [50], a packet manipulation library that supports collected firmware of mainstream IoT manufacturers and used
more than 140 common protocols. Given the parse result the contained protocol binaries as the closed-source targets.
with detailed field type and value along with the structure Compared Fuzzers and Manual Configurations. Since
information (e.g., the hierarchical relationship of a packet and B LEEM is a black-box fuzzer, we select three famous black-
its contained frames), we abstract each packet by retaining box protocol fuzzers widely used in academia and industry
the enumeration field values while maintaining the packet as baselines for black-box schemes, including Peach [18],
structure. Then we construct the abstract packet sequence by BooFuzz [33], and Snipuzz [20]. Also, to demonstrate the ef-
concatenating adjacent abstract packets with the same source. fectiveness of B LEEM, we pick AFLNet [45] and SGFuzz [8],
Based on this process, we implement the state trace construc- two state-of-the-art grey-box protocol fuzzers integrating cov-
tion following the construction rule in §4. Furthermore, to erage and state feedback, for comparison.
demonstrate the scalability of B LEEM, we also selected two We use the corresponding server utilities of the selected
protocols, i.e., QUIC and SSH, that have not been supported SUTs to evaluate the compared protocol fuzzers (these fuzzers
by Scapy for the experiment, as shown in §6.2. One of the can only test one side of the protocol at a time, and they usu-
authors completed coarse-grained Scapy extensions for these ally focus on the server side). To adapt these fuzzers to the
two protocols in about six hours. selected subjects, we followed their tutorials [5, 10, 18, 30, 33]
Guided Fuzzing Module. First, we implement the pro- to prepare the required configurations: (i) For generation-
posed mutation operators that work on two levels: (i) to sup- based fuzzers BooFuzz and Peach, we offered the required
port the packet-level mutation, we design and implement mu- test model for each implementation by modeling the corre-
tation strategy for the five general field types used in Scapy, sponding session embedded in the dialog of the SUT. Specifi-
as summarized in Appendix A.1; and (ii) we perform the cally, in the test model, we depicted the format of the involved
sequence-level mutation based on the Oracle Map, which protocol messages (data model) and their sequencing over the
caches an intermediate map between each abstract packet and session (state model). (ii) For mutation-based fuzzers Snipuzz,
its corresponding recent concrete packet. These two mutation SGFuzz, and AFLNet, we prepared the initial seed corpus
operators are applied in the packet instantiation sub-module. based on the messages captured by executing the SUT. We
Second, following the rule in §5.2, we implement the run- evaluated SGFuzz on partial TCP-based protocol implementa-
time construction of the SSTG by continuously merging ab- tions due to the limitation of the netdriver it bases on and the
stract packet sequences and refining the priority of transitions. in-process requirement of its base fuzzer Libfuzzer [10]. We
Based on it, we implement the sequence generation strategy also extended AFLNet to recognize the protocols that have
proposed in §5.3 for efficient state space traversal. Third, not been supported (e.g., QUIC and PPTP).
the packet instantiation sub-module interacts with the client Experiment Settings. Since the fuzzing performance fluc-
and the server simultaneously and intercepts the exchanged tuates to a certain degree due to the inherent randomness, we
packets. These packets usually contain several layers (e.g., ran each fuzzing tool on each selected project with a 24-hour
network layer and transport layer). To focus on the target pro- time budget and repeated each 24-hour experiment 10 times
tocol’s layer `, we implement the proxy that provides reliable to establish statistical significance of results [35]. For fairness,
communication underlying `. At present, we support layers each fuzzing campaign runs on a Docker container that is
in the network stack. We present the details in Appendix A.2. configured with 1 CPU core and 1G RAM.
 BoringSSL (SSL/TLS) OpenSSL (DTLS) mvfst (QUIC) accel-ppp (PPTP) IEC104 (IEC104)
4000 10000 50000 1500
300
8000 40000
3000
1000 200
6000 30000
2000
Bleem BooFuzz 4000 20000
500 100
1000
1 Peach SGFuzz 2000 10000
AFLNet Snipuzz
0.9 0 0 0 0 0
0:00 6:00 12:00 18:00 24:00 0:00 6:00 12:00 18:00 24:00 0:00 6:00 12:00 18:00 24:00 0:00 6:00 12:00 18:00 24:00 0:00 6:00 12:00 18:00 24:00
Branches Covered

0.8
OpenSSH (SSH) libcoap (CoAP) Dnsmasq (DNS) CycloneDDS (RTPS) Mosquitto (MQTT)
10000 6000 1500 25000
0.7 6000
8000 1 20000
0.6 4000 1000
6000 0.9 15000 4000
0.5
4000 0.8 10000
2000 500 2000
0.4
2000 5000
0.7
0.3 0 0 0 0 0
0:00 6:00 12:00 18:00 24:00 0:00 6:00 12:00 18:00 24:00
0.6 0:00 6:00 12:00 18:00 24:00 0:00 6:00 12:00 18:00 24:00 0:00 6:00 12:00 18:00 24:00
0.2
libiec_iccp_mod (ICCP) GnuTLS (SSL/TLS) 0.5 LibreSSL (SSL/TLS) OpenBGPD (BGP) rudp (RUDP)
5000 4000 6000 120
0.1 2000
4000 0.4 100
3000
0 4000 1500 80
3000
0 0.2 0.4 0.6 0.8 1 0.3
2000 1000 60
2000 0.2 2000 40
1000 500
1000 20
0.1
0 0 0 0 0
0:00 6:00 12:00 18:00 24:00 0:00 6:00 12:00 18:00 24:00
0 0:00 6:00 12:00 18:00 24:00 0:00 6:00 12:00 18:00 24:00 0:00 6:00 12:00 18:00 24:00
0 0.2 0.4 0.6 0.8 1

Time (hh:mm)
Figure 7: The number of unique branches covered (only on the server-side) by different fuzzers on each protocol implementation over ten
24-hour runs. The average number of discovered branches is displayed, alongside with minima and maxima over the individual runs.

6.3 Coverage Analysis ments. This mode modifies the library to disable randomness
and thus is more friendly to traditional fuzzers. Still, B LEEM
Since the compared fuzzing schemes are pretty diverse, we covers more branches than prior techniques on BoringSSL.
need a uniform metric for a fair comparison. Branch coverage Answer to RQ1. Overall, B LEEM can achieve higher cover-
is a commonly used metric to measure the effectiveness of age than existing protocol fuzzers, which means that B LEEM
fuzzers in software testing. Therefore, we use branch cover- can test protocol implementations broadly and deeply.
age as the metric for comparison and utilize LLVM’s Sani-
tizerCoverage [36] to count the number of unique branches 6.4 Bug-Detection Capability
covered by each fuzzer on a target program. To measure the bug detection capability, we adapt B LEEM
Figure 7 shows the branches covered on the server side to fuzz real-world protocol implementations, including open-
by different fuzzers. On average, B LEEM achieved 40.3%, source and closed-source.
35.7%, 23.4%, 48.9%, and 28.5% higher branch coverage Open-Source Targets. We use the number of unique vul-
than Snipuzz, AFLNet, SGFuzz, BooFuzz, and Peach, respec- nerabilities reported by AddressSanitizer [51] and Undefined-
tively, within 24 hours. All results are statistically significant BehaviorSanitizer [13] (a.k.a., ASan and UBSan) as the uni-
according to the Mann–Whitney U test, as recommended form metric. The reason is that the vulnerability detection
by Klees et al. [35]. In 64 out of 69 times, the minimum methods of B LEEM and other fuzzers vary. For example, the
branches achieved by B LEEM exceed the maximum branches classic black-box fuzzer Peach typically detects vulnerabil-
of the prior approaches. In other words, even the worst run ities by checking the liveness of under-test service via port
of B LEEM performs better than the best run of other prior probing. However, not all the vulnerabilities (e.g., some buffer-
approaches, except for partial runs on Dnsmasq, IEC104, and overflow vulnerabilities) will crash the program. Therefore,
OpenBGPD. The reason is that these protocols employ simple we utilize ASan and UBSan to enhance the target program and
interactive logic and packet format. Hence, other fuzzers also use the crashes identified by different fuzzers as the metric
did well on it. Even so, B LEEM also outperforms them on the to represent their vulnerability detection ability. Furthermore,
average branches achieved. DTLS and SSH employ random some Sanitizer-reported crashes may result from the same
nonces to provide replay protection [4, 39], as mentioned in root cause. To eliminate duplicate entries, we utilize the stack
§1. When adapted to their implementations, i.e., OpenSSL and traces in the Sanitizer report for bug deduplication and only
OpenSSH, existing protocol fuzzers have difficulty in complet- consider unique vulnerabilities.
ing the handshake, thus covering fewer branches. In contrast, B LEEM has detected 15 new vulnerabilities in several ex-
B LEEM can penetrate deeply into the protocol logic by utiliz- tensively used implementations of well-known protocols, with
ing the intercepted interactive traffic and the proposed SSTG. 10 CVE identifiers assigned after a coordinated disclosure.
TLS also employs this mechanism, and the results of other We also tried to reproduce these bugs using the other fuzzers
fuzzers on BoringSSL are fine. The reason is that we enabled based on the similar configuration construction method men-
the fuzzer mode provided by BoringSSL [11] in the experi- tioned in §6.2. Table 1 summarizes the vulnerabilities exposed
by B LEEM and whether other fuzzers can find them. Specif- state traces discovered during the SSTG construction, and the
ically, Peach, BooFuzz, AFLNet, SGFuzz, and Snipuzz can column “Len.” indicates the average length of these paths.
only expose 8, 5, 6, 7, and 5 bugs, a strict subset of the bugs The column “Types” indicates the number of different types
uncovered by B LEEM. These protocol implementations have of abstract packets (after concatenation), which are the ele-
been thoroughly tested, and some of them, e.g., GnuTLS [1] ments of the SUT States. The columns “Nodes” and “Trans.”
and LibreSSL [2], have even been incorporated into the OSS- indicate the state and state transition numbers of the SSTG,
Fuzz [27], which demonstrates the effectiveness of B LEEM respectively. The “Branch Coverage” shows the achieved
in bug detection. Some of these bugs are hard to trigger, and unique branches of the whole SUT, including the coverage
we provide the bug details and a case study in Appendix C. achieved on both sides, which is different from §6.3 because
Table 1: Previously unknown vulnerabilities exposed by both B LEEMRand and B LEEM can test the whole system.
B LEEM and the statistics of the compared fuzzers Note that the “Paths” and “Len.” are not necessarily propor-
Subject Type AFLNet Snipuzz SGFuzz BooFuzz Peach B LEEM CVE ID tional to the complexity of the constructed SSTG, as different
LibreSSL Stack Buffer Overflow CVE-2021-41581
GnuTLS Null Pointer Dereference CVE-2021-4209 transition ways of existing nodes and transitions can trigger a
BoringSSL SIGPIPE -
accel-ppp Stack Buffer Overflow CVE-2021-42870 new path while no new nodes or transitions will be found. The
accel-ppp Stack Buffer Overflow CVE-2021-42054
accel-ppp Memory Leak - table shows that the overall unique paths are finite, indicat-
IEC104 Stack Buffer Overflow CVE-2020-20486
IEC104 Segmentation Violation CVE-2020-18731 ing that our provision for the SSTG construction effectively
rdup Memory Leak CVE-2020-20665
libiec_iccp_mod Heap Buffer Overflow CVE-2020-20490 avoids state space explosion. To further illustrate this, we
libiec_iccp_mod Heap Buffer Overflow CVE-2020-20662
libiec_iccp_mod Heap Buffer Overflow CVE-2020-20663 also tried constructing a long initial sequence using the pro-
OpenBGPD Undefined Behavior -
OpenBGPD Undefined Behavior - vided client utility when testing the Dnsmasq, libcoap, and
mvfst Heap Buffer Overflow -
SUM 6 5 7 5 8 15 10 CVEs Mosquitto. The results demonstrate that the path numbers
Closed-Source Targets. We collected four firmware con- achieved on these projects are also within an acceptable range.
taining vulnerable protocol implementations, as disclosed by Table 3: Statistics about the constructed SSTG and the unique
the CVE dataset [16], from different mainstream IoT manufac- branches achieved by B LEEM and B LEEMRand.
turers to evaluate the performance of the selected black-box SSTG Construction SSTG Metrics Branch
Subject Fuzzer
Paths Len. Types Nodes Trans. Coverage
fuzzers in discovering severe vulnerabilities. These CVEs B LEEMRand 42 4.11 32 72 84 4293
BoringSSL
seriously threaten various devices and are classified as CRIT- B LEEM 75 3.82 69 152 183 4549
B LEEMRand 266 6.84 73 247 397 10512
OpenSSL
ICAL by CVSS 3.x Severity and Metrics (see Table 4 in B LEEM 256 5.96 90 267 442 10614
B LEEMRand 2352 7.83 492 1494 2806 53942
Appendix B). We compared B LEEM against selected black- mvfst
B LEEM 10781 7.99 671 2779 6581 55575
B LEEMRand 101 6.07 14 33 46 1384
box fuzzers and used the network-related monitors to de- accel-ppp
B LEEM 30 4.62 11 25 31 1385
tect crashes by checking the liveness of under-test services IEC104
B LEEMRand 39 5.57 84 113 137 279
B LEEM 49 5.86 88 149 164 321
through port probing. We use the time to first crash as the met- OpenSSH
B LEEMRand 43 5.23 20 59 82 12444
B LEEM 97 5.58 30 104 171 14579
ric to evaluate the bug-detection capability of these fuzzers. B LEEMRand 10013 85.08 325 1340 3400 8292
libcoap
B LEEM 10286 83.42 331 1427 4143 8530
As shown in Table 2, B LEEM achieves the best CVE discovery B LEEMRand 2958 22.60 60 163 511 1271
Dnsmasq
performance compared to other fuzzers. B LEEM and Peach B LEEM 1783 14.62 58 155 413 1292
B LEEMRand 55 4.55 18 67 132 22912
CycloneDDS
can find all of these CVEs, while BooFuzz and Snipuzz can B LEEM 139 4.26 31 153 303 23710
B LEEMRand 19522 15.67 215 1037 2815 9284
find only 3 and 1, respectively. On average, B LEEM can find Mosquitto
B LEEM 20652 13.09 253 1085 3142 10285
B LEEMRand 116 9.05 28 96 155 6059
a crash at least 7.5×, 13.3×, and 87.1× faster than Peach, libiec_iccp_mod
B LEEM 314 9.71 41 139 294 6265
BooFuzz, and Snipuzz, respectively, demonstrating B LEEM’s GnuTLS
B LEEMRand 50 4.66 25 70 98 5057
B LEEM 57 4.12 38 92 114 5222
efficiency boost over the state-of-the-art. LibreSSL
B LEEMRand 209 3.93 67 248 394 5473
B LEEM 196 3.78 100 277 385 6157
Table 2: Average time to expose published CVEs B LEEMRand 222 17.73 35 92 131 2072
OpenBGPD
B LEEM 253 17.21 43 111 169 2086
CVE ID Protocol Snipuzz BooFuzz Peach B LEEM B LEEMRand 149 5.86 21 82 151 112
CVE-2018-5767 HTTP - 25min 34min 6min rudp
B LEEM 154 5.70 30 109 185 115
CVE-2020-25067 UPnP 26min 36s 47s 33s
CVE-2019-14457 HTTP - - 652min 35min From each row of Table 3, the complexity of our proposed
CVE-2019-1663 HTTP - 501min 307min 72min
SSTG is roughly in positive correlation to the packet types
Answer to RQ2. B LEEM is capable of finding unknown and the covered unique branches, indicating that the proposed
bugs effectively in real-world protocol implementations. SSTG can reflect the inner system execution status of the SUT
in some degree. With the help of the guided sequence genera-
6.5 Effectiveness of Sequence Generation tion strategy, B LEEM achieves 5.7% more unique branches
To evaluate the effectiveness of the guided sequence gen- than B LEEMRand on average, and the improvement on the
eration (§5.3), we implemented B LEEMRand, a variant of server is typically comparable to that on the client since they
B LEEM, in which we replaced it with random sequence selec- are mutually reinforcing. We also note that B LEEMRand per-
tion and maintained the SSTG construction for comparison. forms better on Dnsmasq and accel-ppp in the complexity
Table 3 shows the mean value of each metric across repe- of the implemented SSTG. Through investigation, we found
titions. The column “Paths” indicates the number of unique that the logic of the corresponding SUTs is relatively simple
compared with other subjects. The randomly generated long these two packets should own the same DCID field to identify
packet sequences of B LEEMRand can easily trigger more out- the same connection, and the PN field value x of the former
puts of the SUT parties, resulting in a more complex SSTG. packet and the value y of the latter should satisfy y = x + 2. If
Nonetheless, B LEEMRand is hard to reach deep states in the these two packets are semantically incorrect, the server will
protocol implementation without guided packet sequence gen- reject them without a response. In the 24-hour experiments
eration, thus covering fewer branches than B LEEM. across 10 repetitions on mvfst, we found that all the com-
Case Study. To intuitively illustrate how B LEEM imple- pared fuzzers failed to trigger this behavior. Instead, B LEEM
ments guided fuzzing and its effectiveness, we use the ses- generates packets by resorting to the packets provided by the
sion discovered during fuzzing mvfst as a case study. By protocol parties. Therefore, it can easily guarantee the above
executing the selected SUT of mvfst, B LEEM constructed conditions and trigger this behavior (q4 − q7 ) at an early stage.
an initial SSTG in Figure 6 as the basis. Directed by Algo- Meanwhile, B LEEM discovered 410 more successor states on
rithm 1, B LEEM tried to stress q1 with the packet pattern average based on q4 − q7 . This state region is hard to trig-
b ⊕ σP , yielding the packet pattern sequence [a ⊕ σ◦ , b ⊕ σP ]. ger for other fuzzers, but discovering this region effectively
Then B LEEM instantiated it and triggered the session in Fig- contributes to B LEEM’s branch coverage.
ure 8 (in reality the client and server interact with B LEEM’s q0
a⨁𝜎∘
q1
b⨁𝜎∘
q2
c⨁𝜎∘
q3

packet instantiation sub-module, we omit it to facilitate the C(a) | S(Ø) S(b) | C(a) C(c) | S(b) S(d) | C(c)

b⨁𝜎 "
understanding). The mutated Initial[CRYPTO, ACK] (de- q4
a⨁𝜎∘
q5
e⨁𝜎∘
q6
f⨁𝜎∘
q7

noted as Initial[CRYPTO, ACK]* ) transmitted at 2 trig- C(a) | S(b) S(e) | C(a) C(f) | S(e) S(d) | C(f)

gered retransmission of the client ( 3 , this Initial[CRYPTO] Figure 9: The System State Tracking Graph (SSTG) after
differs from 1 at the concrete level ). Through our investi- introducing the new packet pattern (b ⊕ σP ) on q1 .
gation of the implementation logic, after reading the second Answer to RQ3. The guided sequence generation strat-
Initial[CRYPTO], the server immediately responds ACK egy of B LEEM is able to increase the exposure of different
for 3 (Initial[ACK]) instead of packing ACK and CRYPTO protocol behaviors, thus contributing to fuzzing effectiveness.
frames into one Initial packet like 2 . Then, it invokes
CloningScheduler, a packet scheduler designed to clone exit- 7 Related Work
ing packets that are still outstanding, to derive the following
Initial[CRYPTO] and Handshake[CRYPTO]. In this way, Protocol Fuzzing. Fuzzing has been widely adopted to test
the testing procedure covered the logic of CloningScheduler, protocol implementations [23–25, 38, 61–63]. Existing works
where a known problem has been exposed [19]. focus on individual packet generation and lose sight of con-
As a result, exercising q1 using the new packet pattern textual correctness between packets in a sequence. Peach [18]
b⊕σP achieved a new state trace. Empowered by the feedback and BooFuzz [33] select a state trace in the user-defined state
collector, B LEEM can monitor this automatically and update model each time and separately generate packets for these
it on the initial SSTG with an extended alphabet: states. Scapy fuzzing API [49] generates individual packets
b = {e : Initial[ACK]+Initial[CRYPTO]+Handshake[CRYPTO],
Ω based on given values, which is equivalent to a packet-level
f : Handshake[CRYPTO,ACK]}, mutation operator. Although these approaches provide syn-
yielding the overall SSTG shown in Figure 9. tactically valid packets that prevent early parse error, they
Client Server are incapable of handling the parameter dependency and
①
Initial[CRYPTO] generating correct values for the context-sensitive dynamic
Initial[CRYPTO, ACK]*, fields (e.g., handshake configuration of SSL). Meanwhile,
Handshake[CRYPTO]
②
Initial[CRYPTO]
they are pretty random and do not use feedback to guide
③
Initial[ACK], Initial[CRYPTO], fuzzing. Instead, B LEEM generates packets at a sequence
Handshake[CRYPTO]
④ level. It provides a sequence-based feedback mechanism to
Handshake[CRYPTO, ACK]
⑤ navigate protocol-logic exploration and a sequence-level ma-
1-RTT[APPLICATION_CLOSE]
⑥ nipulation to discover anomalies under out-of-order or du-
plicated packets. Meanwhile, it accounts for the contextual
Figure 8: The mvfst handshake flow if the packet at 2 is information by leveraging the intercepted packets between
mutated (Initial[CRYPTO,ACK]* ). the SUT parties.
Unfortunately, triggering such logic is non-trivial for other Some recent works introduce state awareness for protocol
fuzzers because they need to craft the two Initial[CRYPTO] fuzzing. SGFuzz [8] relies on a programmatic intuition that
packets carefully: (i) The two packets need to be syntactically the state variables used in protocol implementations encode
correct, which can be easily guaranteed by generation-based fine-grained program processing actions and often appear in
fuzzers like Peach but is hard for mutation-based fuzzers like enum-type variables. It recognizes these variables, injects
AFLNet. (ii) They need to correctly set some parameters instrumentation to monitor their assignment, and uses their
in the packets to ensure semantic correctness. For example, different values to identify different server program states.
In comparison, without instrumentation, B LEEM leverages derive the SSTG structure from the observed network traffic
the enum in the packet fields to identify packet type and to identify the state space and guide fuzzing. Meanwhile, the
constructs the states by combining the packet types with bidi- SSTG encompasses all protocol parties, unlike the traditional
rectional communication information. We also provide an state machine that depicts only one protocol party.
example in Appendix D. StateAFL [43] adopts fine-grained
compile-time instrumentation to obtain runtime information 8 Discussion
for protocol state inference. Nyx-Net [48] solves the re- Despite B LEEM’s positive results, we briefly discuss limita-
producibility problem mentioned in §1 by ensuring noise- tions and avenues of further improvement.
free fuzzing through a snapshot-based approach. These ap- First, the feedback collector analyzes the output packets
proaches require the source code or the binary of the protocol based on Scapy’s parsing capability. However, some protocols,
implementation and thus do not scale to the black-box nature. especially those proprietary protocols, are not supported by
Instead, B LEEM applies a noninvasive feedback mechanism Scapy. Since B LEEM can capture the network traffic and gain
and models state transitions across packet exchanges, thus can additional traffic beyond the initial session through fuzzing,
be scaled to diverse protocols even in the black-box setting. we can recognize unsupported protocols by resorting to traffic-
Some works utilize the server response to optimize fuzzing. based protocol reverse engineering [12, 57].
AFLNet [45] leverages the status code in the server re- Second, the current representation of the SSTG cannot
sponse as state feedback and uses it to guide packet mutation. always guarantee reproducibility due to its inherent non-
Snipuzz [20] infers the grammatical role of each message deterministic and coarse-grained transition labeling. We can
byte by analyzing the server response using a hierarchical solve this by transforming the SSTG into a deterministic finite
clustering strategy. This strategy works well for IoT proto- automata using typical algorithms [41, 55] and recording fine-
cols whose responses are usually textual and organized in a grained mutation information, such as the detailed subclass
common format such as JSON. These approaches work well and parameters of the mutation operator.
but are tightly coupled with specific protocol formats. In con- Third, B LEEM now supports crash detection and memory-
trast, B LEEM analyzes the semantics conveyed in the output related bugs with ASan/UBSan. B LEEM can also detect se-
to obtain system feedback and can be applied to both textual mantic bugs if provided with corresponding oracles. For exam-
and binary protocols. Meanwhile, it analyzes both client and ple, if provided with the packet-exchange constraints, B LEEM
server output and thus can obtain more information. can detect non-compliance with protocol specification by
State Machine Inference. The most closely related works analyzing the SUT’s state trace. The cause of CVE-2021-
employ learning algorithms to infer protocol state machines, 40523 [42] is that the server may fail to send WILL/WONT
and there are two different technologies. response for WILL commands, which violates the property
Active-inference-based approaches [17, 21–23, 52] actively restricted in RFC 854 [46]. To detect this bug, we can provide
generate packet sequences to query a protocol implementation such a packet-exchange constraint for B LEEM: a WILL packet
and infer a state machine using model learning algorithms, from the client should be followed by a WILL/WONT packet
such as Angluin’s L* algorithm [7]. Applying this algorithm from the server under normal conditions. With the help of
usually requires tailoring a mapper to translate between the the abstract packets, B LEEM can focus on the packet types,
abstract alphabets of the model and the concrete packets of the thus facilitating analysis. Then, it can detect the bug by check-
implementation, which is not reusable for different protocols ing, for the SUT State that matches hS(∗) | C(WILL)i (“*” is a
and implementations. Instead, B LEEM abstracts packets by wildcard), whether its outgoing edge with label WILL⊕σ◦ tran-
automatic semantic extraction and instantiates packets using sitions to state hC(WILL) | S(WILL)i or hC(WILL) | S(WONT)i.
the interactive traffic. If not, a bug exists.
Passive-inference-based approaches [14, 26, 28, 29] infer
a state machine by analyzing a corpus of packet sequences 9 Conclusion
sampled on the network. Pulsar [26] analyzes the sampled In this paper, we present B LEEM, a packet-sequence-oriented
network traces and infers a generative model for message for- protocol fuzzer that applies an evolutionary approach to ex-
mat and protocol states. AutoFuzz [28] analyzes the sampled plore the massive protocol state space: it accesses the sys-
traffic to infer the server’s finite state machine (FSM) and con- tem feedback by analyzing the output sequences and dynami-
ducts server fuzzing based on this stationary FSM. These ap- cally tunes the exploration direction by applying the proposed
proaches perform fuzzing based on the inferred model. There- guided fuzzing strategy. Meanwhile, B LEEM generates highly
fore, their fuzzing effectiveness relies on the completeness of protocol-logic-aware packet sequences by leveraging the ob-
the captured network traffic. Instead, B LEEM constructs the served interactive traffic. Compared to state-of-the-art fuzzers,
initial SSTG based on the SUT dialog and gradually enriches B LEEM can achieve higher coverage and detect more bugs in
it at runtime with the packets generated during guided fuzzing, real-world protocol implementations. B LEEM is fully auto-
which builds a closed-loop of vulnerability detection. matic and can be applied to test the implementations of most
Most of all, we do not infer the protocol’s state machine but general protocols in the black-box setting.
Acknowledgments [14] Paolo Milani Comparetti, Gilbert Wondracek, Christo-
pher Krügel, and Engin Kirda. Prospex: Protocol speci-
We thank the anonymous reviewers for their construc- fication extraction. IEEE Symposium on Security and
tive feedback and suggestions. This research is sponsored Privacy, 2009.
in part by the National Key Research and Development
Project (No. 2022YFB3104000, No2021QY0604) and NSFC [15] CVE-2014-0160. Heartbleed - a vulnerability in openssl.
Program (No. 62022046, 92167101, U1911401, 62021002). 2014. http://heartbleed.com.

[16] US National Vulnerability Database. Common vulnera-

References bilities and exposures (CVE).

[1] OSS-Fuzz/Gnutls. https://github.com/google/ [17] Joeri de Ruiter and Erik Poll. Protocol state fuzzing of
oss-fuzz/tree/master/projects/gnutls. TLS implementations. 2015 USENIX Security Sympo-
sium.
[2] OSS-Fuzz/Libressl. https://github.com/google/
oss-fuzz/tree/master/projects/libressl. [18] Michael Eddington. Peach fuzzing plat-
form. https://gitlab.com/gitlab-org/
[3] RFC 4346. The transport layer security (TLS) proto- security-products/protocol-fuzzer-ce.
col. section f.1.1.2: Rsa key exchange and authentica-
tion. Website. https://www.rfc-editor.org/rfc/ [19] facebookincubator. A known problem in the
rfc4346#appendix-F.1.1.2. CloningScheduler in Facebook mvfst. https:
//github.com/facebookincubator/mvfst/blob/
[4] RFC 6347. Datagram transport layer secu- 421196ec98a9abd69c7a4353c555a0c981a69109/
rity version 1.2. section 4.1.2.6: Anti-replay. quic/api/QuicBatchWriter.cpp#L17.
https://datatracker.ietf.org/doc/html/
rfc6347#section-4.1.2.6. [20] Xiaotao Feng, Ruoxi Sun, Xiaogang Zhu, Minghui Xue,
Sheng Wen, Dongxi Liu, Surya Nepal, and Yang Xi-
[5] aflnet. AFLNet: A greybox fuzzer for network protocols. ang. Snipuzz: Black-box fuzzing of IoT firmware via
https://github.com/aflnet/aflnet. message snippet inference. 2021 ACM SIGSAC CCS.
[6] Pedram Amini and Aaron Portnoy. Sulley. 2012. https: [21] Tiago Ferreira, Harrison Brewton, Loris D’antoni, and
//github.com/OpenRCE/sulley. Alexandra Silva. Prognosis: Closed-box analysis of net-
work protocol implementations. 2021 ACM SIGCOMM.
[7] Dana Angluin. Learning regular sets from queries and
counterexamples. Inf. Comput., 75, 1987. [22] Paul Fiterau-Brostean, Ramon Janssen, and Frits W.
[8] Jinsheng Ba, Marcel Böhme, Zahra Mirzamomen, and Vaandrager. Combining model learning and model
Abhik Roychoudhury. Stateful greybox fuzzing. 2022 checking to analyze TCP implementations. 2016 CAV.
Usenix Security Symposium. [23] Paul Fiterau-Brostean, Bengt Jonsson, Robert Merget,
[9] Domagoj Babic, Stefan Bucur, Yaohui Chen, Franjo Joeri de Ruiter, Konstantinos Sagonas, and Juraj So-
Ivancic, Tim King, Markus Kusano, Caroline Lemieux, morovsky. Analysis of DTLS implementations using
László Szekeres, and Wei Wang. FUDGE: Fuzz driver protocol state fuzzing. 2020 USENIX Security Sympo-
generation at scale. 2019 ACM ESEC/FSE. sium.

[10] bajinsheng. SGFuzz: Stateful greybox fuzzer. https: [24] Matheus E. Garbelini, Vaibhav Bedi, Sudipta Chattopad-
//github.com/bajinsheng/SGFuzz. hyay, Sumei Sun, and Ernest Kurniawan. Braktooth:
Causing havoc on bluetooth link manager via directed
[11] BoringSSL. Fuzzer mode. https://github.com/ fuzzing. 2022 USENIX Security Symposium.
google/boringssl/blob/master/FUZZING.md.
[25] Matheus E. Garbelini, Chundong Wang, Sudipta Chat-
[12] Georges Bossert, Frédéric Guihéry, and Guillaume Hiet. topadhyay, Sumei Sun, and Ernest Kurniawan. Sweyn-
Towards automated protocol reverse engineering using Tooth: Unleashing mayhem over bluetooth low energy.
semantic information. 9th ACM symposium on Informa- 2020 USENIX Annual Technical Conference.
tion, computer and communications security, 2014.
[26] Hugo Gascon, Christian Wressnegger, Fabian Yam-
[13] Clang. Clang 15.0.0 documentation, undefinedbe- aguchi, Dan Arp, and Konrad Rieck. Pulsar: Stateful
haviorsanitizer. https://clang.llvm.org/docs/ black-box fuzzing of proprietary network protocols. Se-
UndefinedBehaviorSanitizer.html. cureComm, 2015.
[27] Google. OSS-Fuzz. https://github.com/google/ [42] MITRE. CVE-2021-40523.
oss-fuzz.
[43] Roberto Natella. StateAFL: Greybox fuzzing for stateful
[28] Serge Gorbunov and Arnold Rosenbloom. Autofuzz: network servers. ArXiv, abs/2110.06253, 2021.
Automated network protocol fuzzing framework. Ijcsns,
[44] OpenBSD. Libressl. https://www.libressl.org.
2010.
[45] Van-Thuan Pham, Marcel Böhme, and Abhik Roychoud-
[29] Yating Hsu, Guoqiang Shu, and David Lee. A model-
hury. AFLNET: A greybox fuzzer for network protocols.
based approach to security flaw detection of network
2020 ICST.
protocol implementations. IEEE International Confer-
ence on Network Protocols, 2008. [46] J. Postel and J. Reynolds. Rfc854, telnet proto-
col specification. https://datatracker.ietf.org/
[30] Immor278. Snipuzz-py. https://github.com/ doc/html/rfc854.
Immor278/Snipuzz-py.
[47] Postel, J. and J. Reynolds. File transfer protocol, 1985.
[31] Kyriakos K. Ispoglou, Daniel Austin, Vishwath Mohan, https://www.rfc-editor.org/rfc/rfc959.html.
and Mathias Payer. FuzzGen: Automatic fuzzer genera-
tion. 2020 USENIX Security Symposium. [48] Sergej Schumilo, Cornelius Aschermann, Andrea Jem-
mett, Ali Reza Abbasi, and Thorsten Holz. Nyx-net: net-
[32] Iyengar, J., Ed., and M. Thomson, Ed. QUIC: A udp- work fuzzing with incremental snapshots. Seventeenth
based multiplexed and secure transport, 2021. https: European Conference on Computer Systems, 2022.
//www.rfc-editor.org/rfc/rfc9000.html.
[49] secdev. Scapy Fuzzing API. https:
[33] jtpereyda. BooFuzz: Network protocol fuzzing for hu- //scapy.readthedocs.io/en/latest/usage.
mans. https://github.com/jtpereyda/boofuzz. html#fuzzing.
[34] Jinho Jung, Stephen Tong, Hong Hu, Jungwon Lim, [50] secdev. Scapy: Packet crafting for python2 and python3.
Yonghwi Jin, and Taesoo Kim. WINNIE: Fuzzing https://scapy.net.
windows applications with harness synthesis and fast
cloning. 2021 NDSS. [51] Kostya Serebryany, Derek Bruening, Alexander
Potapenko, and Dmitriy Vyukov. AddressSanitizer:
[35] George Klees, Andrew Ruef, Benji Cooper, Shiyi Wei, A fast address sanity checker. 2012 USENIX Annual
and Michael W. Hicks. Evaluating fuzz testing. 2018 Technical Conference.
ACM SIGSAC CCS.
[52] Juraj Somorovsky. Systematic fuzzing and testing of
[36] LLVM. SanitizerCoverage. https://clang. TLS libraries. 2016 ACM SIGSAC CCS.
llvm.org/docs/SanitizerCoverage.html#
edge-coverage. [53] Synopsis. Defensics fuzz testing.

[37] Zhengxiong Luo, Feilong Zuo, Yu Jiang, Jian Gao, Xun [54] Peach Tech. Peach fuzzer configuration file (Peach
Jiao, and Jiaguang Sun. Polar: Function code aware fuzz Pit). Website. https://peachtech.gitlab.io/
testing of ICS protocol. ACM Trans. Embed. Comput. peach-fuzzer-community/v3/PeachPit.html.
Syst., 18:93:1–93:22, 2019. [55] Ken Thompson. Programming techniques: Regular ex-
[38] Zhengxiong Luo, Feilong Zuo, Yuheng Shen, Xun Jiao, pression search algorithm. Communications of the ACM,
Wanli Chang, and Yu Jiang. ICS protocol fuzzing: Cov- 1968.
erage guided packet crack and generation. ACM/IEEE [56] Twistedmatrix. Twisted: building the engine of your
Design Automation Conference (DAC), 2020. network. https://twistedmatrix.com.
[39] T. Kohno M. Bellare and C. Namprempre. The secure [57] Yapeng Ye, Zhuo Zhang, Fei Wang, X. Zhang, and
shell (SSH) transport layer encryption modes. https: Dongyan Xu. NetPlier: Probabilistic network protocol
//datatracker.ietf.org/doc/html/rfc4344. reverse engineering from message traces. 2021 NDSS.
[40] Kenneth L. McMillan and Lenore D. Zuck. Formal [58] Michal Zalewski. American fuzzy lop. 2015.
specification and testing of QUIC. ACM Special Interest
Group on Data Communication, 2019. [59] Cen Zhang, Xingwei Lin, Yuekang Li, Yinxing Xue,
Jundong Xie, Hongxu Chen, Xinlei Ying, Jiashui Wang,
[41] Robert McNaughton and Hisao Yamada. Regular ex- and Yang Liu. APICraft: Fuzz driver generation for
pressions and state graphs for automata. IRE Trans. closed-source SDK libraries. 2021 USENIX Security
Electron. Comput., 9, 1960. Symposium.
[60] Mingrui Zhang, Jianzhong Liu, Fuchen Ma, Huafeng deletion, element-addition (based on the Oracle Map cor-
Zhang, and Yu Jiang. IntelliGen: Automatic driver syn- pus), and list-order-shuffle on the original list.
thesis for fuzz testing. 2021 ICSE-SEIP.
• The EnumField represents the field whose possible values
[61] Yong-Hao Zou, Jia-Ju Bai, Jielong Zhou, Jianfeng Tan, are taken from a given enumeration. For example, in HTTP,
Chenggang Qin, and Shih-Min Hu. TCP-Fuzz: De- the Method field with possible values [“GET”, “POST”,
tecting memory and semantic bugs in TCP stacks with “HEAD”, ...] is EnumField. The EnumField mutation op-
fuzzing. 2021 USENIX Annual Technical Conference. erator selects a value from the valid enumeration set with
high probability. It also provides value out of the valid set,
[62] Feilong Zuo, Zhengxiong Luo, Junze Yu, Ting Chen,
with low probability, to manifest corner cases.
Zichen Xu, Aiguo Cui, and Yu Jiang. Vulnerability
detection of ICS protocols via cross-state fuzzing. IEEE
Trans. Comput. Aided Des. Integr. Circuits Syst., 2022.

[63] Feilong Zuo, Zhengxiong Luo, Junze Yu, Zhe Liu, and A.2 Packet Instantiation Sub-Module
Yu Jiang. PAVFuzz: State-sensitive fuzz testing of pro-
tocols in autonomous vehicles. ACM/IEEE Design Au- For target protocols on different layers, we implement cor-
tomation Conference (DAC), 2021. responding proxies working on underlying layers to provide
reliable underlying communication, including TCP Proxy,
A Implementation Details UDP Proxy, IP Proxy, and Ether Proxy. For example, when
fuzzing SSL protocol, we can use a TCP Proxy that provides
The implementation is well-modularized. We defined uni- reliable TCP connections with the client and server so that
form interfaces between the main fuzzing process and each we can focus on fuzzing the SSL packet instead of the full
module to facilitate scalability. In this way, we can easily ex- protocol stack. To this end, for a target protocol on layer `, the
tend B LEEM with new mutation operators, customized moni- support proxy working on ` − 1 should: (i) provide network
tors (e.g., a semantically-aware monitor), and new protocol isolation on layer ` − 1 to intercept the exchanged payload
stacks (e.g., Bluetooth stack) by implementing corresponding of layer `; (ii) maintain the two links with the client and
required interface functions. server and synchronize their status; and (iii) support efficient
concurrent interaction with the client and server on layer `.
A.1 Packet-Level Mutation Operators We implement the first requirement by configuring the
client with a proxy-provided service address, which differs
Scapy has identified five general field types, including from the server’s service address. For example, for the target
NumberField, StringField, ListField, EnumerationField, and SSL server listening on TCP port 4433, we start TCP Proxy
LengthField. We devise respective mutation operators for and bind to TCP port 4432, and then configure the client with
these field types based on their features. More specifically, the service address of TCP port 4432. This step is noninvasive
• The NumberField mutation operator performs random ad- since the client is typically configurable regarding the service
dition or subtraction operations to the original value while address to connect.
considering the valid value range or returns a number ran-
The second requirement is tailored for the TCP Proxy since
domly selected from the valid value range.
TCP is connection-oriented. For target protocol running on
• The LengthField holds the length value of referenced field. TCP, the TCP Proxy is responsible for maintaining the two
The LengthField mutation operator inherits all the opera- TCP links with the client and server and synchronizing their
tions defined in the NumberField mutation operator. It also status, including the connection establishment and connection
provides an additional operation of replacing the original close. We implement it upon Twisted [56], which provides an
value with extreme values (e.g., zero, negative values, the event-driven programming paradigm for internet applications.
maximum, and the minimum) to manifest corner cases that Specifically, we design and implement two Protocols based
cause memory errors, since their values usually affect the on twisted.internet.protocol.Protocol to handle the
memory access in the program (as the bug case study given data of these two links in an asynchronous manner.
in Appendix C shows).
Third, to concurrently handle bidirectional traffic, we im-
• The StringField mutation operator conducts a finite plement asynchronous interaction logic to separately manage
combination of string-splicing, substring-duplication, and the traffic of two directions, i.e., the server-to-client traffic
substring-deletion on the original string. and client-to-server traffic. The interaction logic of each direc-
• The ListField depicts the field holding a list with items tion continuously sniffs the traffic, conducts mutation on the
of the same type. The ListField mutation operator per- received packet (as directed by the proposed strategy), and
forms a finite combination of element-duplication, element- sends out the mutated packet.
 Table 4: Published CVE IDs of protocol implementations in firmware and the emulation setting in our experiment

CVE ID Device Type Vendor Model Firmware Version Vulnerable Binary Protocol Emulation Platform
CVE-2018-5767 Router TENDA AC15 15.03.1.16 bin/httpd HTTP QEMU user-mode
CVE-2020-25067 Router NETGEAR R8300 1.0.2.130 usr/sbin/upnpd UPnP QEMU full-system
CVE-2019-14457 IP Camera VIVOTEK CC8160 0100d usr/sbin/httpd HTTP QEMU user-mode
CVE-2019-1663 Router CISCO RV130 1.0.3.44 usr/sbin/httpd HTTP QEMU full-system

A.3 Monitors and selected SUT. To facilitate a fair comparison, the selected
SUT (cf. the fourth column) all come from off-the-shelf utili-
We design and implement Process Monitor and Network Mon-
ties provided in the project, except for the client utility used
itors to detect crashes. These monitors can be used in combi-
to test accel-ppp and Dnsmasq, where we resorted to related
nation according to different scenarios.
utilities in the Linux community.
Process Monitor. When the target process is locally ac-
cessible, the Process Monitor detects crashes by checking
whether the process was terminated by a system signal (e.g., C Bug Case Study and Coordinated Disclosure
SIGSEGV). We can also enhance the program with ASan to
detect memory corruption (in this case, the process will be Table 6 shows the detailed information on previously un-
terminated by SIGABRT when a memory error occurs). known vulnerabilities exposed by B LEEM.
Network Monitor. We implement the following network 1 char working [ DOMAIN_MAX_LEN + 1] = { 0 };
monitors, which can be employed locally or remotely. 2 size_t i , wi = 0;
3 for (i = 0; i < len ; i ++) {
• TCP Monitor. For the TCP-based protocols, we detect 4 char c = candidate [i ];
crashes by trying to conduct a TCP connection to the lis- 5 ... // checking validation of candidate [i]
6 if ( wi > DOMAIN_MAX_LEN ) // bug : the corner case is
tening port since TCP is connection-oriented. mistakenly handled
• UDP Echo Monitor. For the UDP-based protocols, this 7 goto bad ;
8 working [ wi ++] = c;
monitor provides an interface that allows users to identify 9 if (i == len - 1) {
the heartbeat packet of the under-test protocol. Then it 10 candidate_domain = strdup ( working ); // buffer may
lack ’\0 ’ termination when invoking strdup
works by assembling it with a UDP header, sending it to 11 }
the listening port, and waiting for a response. 12 }

• Ping Monitor. The Ping Monitor validates a target is still

alive by performing ICMP Echo tests. Figure 10: Simplified code snippets related to CVE-2021-41581.

Case Study of CVE-2021-41581. This bug is revealed in

B Selected Targets in Evaluation the LibreSSL [44], the default TLS provider for OpenBSD
and macOS. It is triggered at the certificate validation stage.
Table 4 shows the information on the selected closed-source
Figure 10 presents the relevant code snippets of the flawed
targets, including the firmware information, corresponding
function, which walks through a given buffer candidate with
vulnerable protocol binaries, and the emulation settings in our
length len, and checks its validation as a mailbox. In this
experiment.
process, the buffer working is firstly cleared for initializa-
Table 5: Selected open-source protocol implementations tion (Line 1) and then is used to cache each checked character
Protocol Subject LOC SUT (client, server) c in candidate (Lines 4, 8). Actually, the developers have
TLS/SSL BoringSSL 92K tools/bssl client, tools/bssl server considered the overflow but mistakenly handled the corner
DTLS OpenSSL 884K apps/openssl s_client, apps/openssl s_server case (Line 6). If the len is equal to DML, the original ‘\0’
QUIC mvfst 104K samples/echo client, samples/echo server
PPTP accel-ppp 61K pptpsetup (Linux utility), sbin/accelpppd termination of working would be overwritten when the vari-
IEC104 IEC104 3K iec104_monitor, iec104_monitor able wi equals DML (line 8), leading to the stack-based buffer
SSH OpenSSH 136K ssh, sshd
CoAP libcoap 28K examples/coap-client, examples/coap-server over-read when invoking strdup (Line 10) due to lack of ‘\0’
DNS Dnsmasq 32K nslookup (Dnsutils, a Linux utility), dnsmasq termination. Triggering this bug is non-trivial because the
RTPS CycloneDDS 226K HelloworldPublisher, HelloworldSubscriber
MQTT Mosquitto 45K mosquitto_sub & mosquitto_pub , mosquitto packet sequence should satisfy the following three conditions:
ICCP libiec_iccp_mod 74K client_example2, server_example1 (i) the program logic enters the certificate validation stage; (ii)
RUDP rudp 570 rudp.exe, rudp.exe
TLS/SSL LibreSSL 596K apps/openssl s_client, apps/openssl s_server the length len of the buffer candidate should be DML; and
TLS/SSL GnuTLS 429K src/gnutls-cli, src/gnutls-serv (iii) the characters in the buffer candidate should be valid
BGP OpenBGPD 12K bgpd, bgpd
to pass the check (Line 5). Otherwise, it would be rejected at
Table 5 shows detailed information on target open-source the early process, and this case cannot be covered. If this bug
protocol implementations, including the protocol type, pro- is exploited, the devices that run this protocol may expose
tocol implementation name, lines of implementation code, sensitive data. The three state-of-the-arts all failed to expose
 Table 6: Previously unknown vulnerabilities exposed by B LEEM
Protocol Subject Information Status CVE ID CVSS Score
TLS/SSL LibreSSL Stack buffer overflow in the x509_constraints_parse_mailbox in libcrypto/x509/x509_constraints.c Assigned CVE-2021-41581 5.5 MEDIUM
TLS/SSL GnuTLS MD_UPDATE does not prohibit zero-length input from illegal address, causing null pointer dereference Assigned CVE-2021-4209 6.5 MEDIUM
TLS/SSL BoringSSL The server tries to write to a socket that has been shut down even after the client has sent TLS Alert Fixed - -
PPTP accel-ppp Stack-based out-of-bounds read in post_msg when processing a call_clear_request Assigned CVE-2021-42870 7.5 HIGH
PPTP accel-ppp Stack-based out-of-bounds read in the server if the client exits after authentication Assigned CVE-2021-42054 7.5 HIGH
PPTP accel-ppp Memory allocated to pool in ippool_init2 is not free when exits, which can cause a denial of service Requested - -
IEC104 IEC104 Stack buffer overflow in the parameter Iec10x_Sta_Addr Assigned CVE-2020-20486 7.5 HIGH
IEC104 IEC104 Segmentation violation in the Iec104_Deal_FirmUpdate function Assigned CVE-2020-18731 7.5 HIGH
UDP rdup Memory leak in the rudp_delete function because memory allocated rudp is not freed Assigned CVE-2020-20665 7.5 HIGH
ICCP libiec_iccp_mod Heap buffer overflow in the ByteStream_readOctets in common/byte_stream.c Assigned CVE-2020-20490 7.5 HIGH
ICCP libiec_iccp_mod Heap buffer overflow in the parseIncomingMessage in mms/iso_cotp/cotp.c Assigned CVE-2020-20662 6.5 MEDIUM
ICCP libiec_iccp_mod Heap buffer overflow in the MmsConnection_create in mms_client_connection.c Assigned CVE-2020-20663 6.5 MEDIUM
BGP OpenBGPD Undefined behavior, a null pointer is passed as the first argument of memcpy in imsg.c Fixed - -
BGP OpenBGPD Undefined behavior, an incorrect bitwise shift in imsg.c Fixed - -
QUIC mvfst Heap buffer overflow in the EchoServerTransportFactory::make function in EchoServer.h Requested - -
SUM 15 10 CVEs -

Change Cipher Spec (20) Hello Request (0)

this bug because they have a small probability of success in Content Type Version
Alert (21) Client Hello (1)
entering this deep protocol state. Instead, our approach resorts Handshake (22)
Length Handshake Type Server Hello (2)

to the packets provided by the SUT parties and applies guided Application Data (23)
Other Fields
Certificate (11)
…
fuzzing, thus can easily satisfy (i) and (iii), allowing for more
Value of the enum field in the packet Other valid values of the enum field
efforts in the exploration of (ii).
We responsibly disclosed the vulnerabilities we found. Be- Figure 11: Enumeration fields in an SSL Server Hello packet
fore publicly disclosing our findings, we reported the vul-
nerabilities to the respective vendors following their security SGFuzz directly uses the different values of the state vari-
procedures and coordinated appropriate disclosure periods ables to identify different program states and captures state
with them, which aligns with industry standards. No vendor transitions by monitoring the assignment of recognized state
required us to redact our results prior to paper submission. variables based on instrumentation.
Correspondingly, Figure 11 shows an SSL Server Hello
D Enumeration-Type State Variable and Field packet. The Content Type and Handshake Type fields are
of type enumeration, and the Version and Length fields are
For BoringSSL, SGFuzz recognizes several state variables, in- of type number and length, respectively. These enumeration-
cluding tls12_server_hs_state_t and ssl_shutdown_t type fields determine the packet type and indicate the protocol
as follows. These variables encode fine-grained program state. Specifically, the Content Type field, with four valid
processing actions, including protocol-state-related (e.g., values, is set to Handshake (22), and the Handshake Type
tls12_server_hs_state_t) and implementation-logic- field is set to Server Hello (2), indicating this is a Server
related (e.g., ssl_shutdown_t). For example, the server Hello packet exchanged during the handshake phase. Other
program uses the tls12_server_hs_state_t to repre- fields, such as Version and Length, own low association
sent the protocol handshake state. It sets this variable to with the protocol state. Hence, B LEEM abstracts this packet
state12_send_server_hello when generating and send- to Handshake[Server_Hello] by omitting other fields. Ac-
ing a Server Hello packet. Meanwhile, it uses the variable tually, from the program’s perspective, when a client or server
ssl_shutdown_t to represent the program shutdown state receives a packet, it also first determines the packet type by
for the read half of the connection. If the server is configured parsing these fields and then takes corresponding actions (in-
not to send a close_notify packet, it sets this variable to cluding setting state variables) according to the state machine.
ssl_shutdown_close_notify, indicating doing nothing. Without instrumentation, B LEEM utilizes an on-the-fly ap-
enum tls12_server_hs_state_t { proach. Since the enumeration fields typically identify the
state12_start_accept = 0,
state12_read_client_hello , packet type and thus indicate the protocol state, B LEEM uses
state12_read_client_hello_after_ech , this key information to abstract concrete packets and con-
state12_select_certificate ,
state12_select_parameters , structs the SUT States by combining the abstract packets with
state12_send_server_hello , bi-directional communication information, as shown in §4.
...
};

enum ssl_shutdown_t {
ssl_shutdown_none = 0,
ssl_shutdown_close_notify = 1,
ssl_shutdown_error = 2,
};