Pci e Error Rate
Pci e Error Rate
Abstract— Modern high speed serial buses are generally better understand the real world performance impact of
required by specification to achieve a maximum bit error ratio. increasing error rates beyond the specification level.
Are these requirements too restrictive? This paper will look at
a series of studies on Peripheral Component Interconnect TABLE I. BER SPECIFICATIONS FOR SOME HIGH SPEED BUSES
Express and Serial AT Attachment, investigating the impact of
Link BER Spec
bit error ratio on bus performance. The results of these studies
suggest that typical bit error ratio requirements may be PCI Express Gen 3 10-12 [1]
conservative. The findings suggest that alternative bus 10 Gigabit Ethernet 10-12 [2]
performance specifications should be considered that would SATA 3.x 10-12 [3]
open new possibilities for design, validation and
USB 3.x 10-12 [4]
manufacturing test tradeoffs.
Keywords-bit error ratio; BER; electrical validation; high These studies are interesting in that they provide some
speed interconnect; high speed bus; I/O. data justifying room for design tradeoffs. For example, there
may be significant cost savings opportunities, trading off
I. INTRODUCTION slight performance impact for lower cost material. Consider
Modern high speed serial bus specifications generally the example of a system design with long Peripheral
have a requirement for maximum Bit Error Ratio (BER) Component Interconnect Express (PCIe) bus routing lengths.
[1][2][3][4]. In this context, bit error ratio is defined as the Instead of using more expensive low loss Printed Circuit
fraction of bits transmitted over the high speed interconnect Board (PCB) material, it may make sense to sacrifice error
that are interpreted incorrectly at the receiving device—i.e., a rate and realize a cost savings with standard FR4 PCBs.
bit originally transmitted as a “1” is interpreted as a “0” or Likewise, there may be power reduction opportunities for
vice-versa. Table I summarizes these for a variety of buses: low power devices, trading off performance for lower power
Third Generation Peripheral Component Interconnect operation, without sacrificing data integrity.
Express (PCIe Gen 3), 10 Gigabit Ethernet, Serial AT It is easy to show through either empirical measurements
Attachment (SATA), and Universal Serial Bus (USB). Note or theoretical arguments that the bus BER of a product is a
that there is no inherent need or expectation that each distribution when measured across multiple instances of that
interface type has the same BER requirement, but the table product. Factors that induce this distribution include,
illustrates that 10−12 is quite commonly used. among other things, the variations in the characteristics of
Many high speed buses such as the ones listed in Table I the board interconnects, receiver circuitry, and transmitter
utilize error detection schemes such as a Cyclical circuitry. For example, in the voltage domain of the bus
Redundancy Check (CRC) at the receiving device to detect signal, these factors lead to a distribution of the electrical
any signal integrity-induced bit errors that could have margin, Vm, where Vm represents the amount of voltage
occurred over the interconnect. In such a scheme, in the swing at the receiving device beyond the minimum required
event of a detected error, a request is sent to the transmitting voltage detection threshold.
device to send the data again (a retry). Ideally, a target BER To simplify the example, consider an ideal case where
level on an interconnect that employs a CRC check must there is no noise in the system when Vm is measured, i.e.,
take into account both the effectiveness of the CRC scheme Vm is the noise-free voltage signal margin. In a real system,
with respect to the protected data packet size as well as the bit errors result from noise adding or subtracting from this
performance losses that result from error-induced retries on margin. In a zero-mean additive white Gaussian noise
the bus. Although studies and publications on the model such as that described in [8], the Vm distribution may
effectiveness of CRC error detection have occurred for
be mapped into a BER distribution via the following
multiple decades [5][6], as far as the authors know, there
relationship:
have been few, if any, studies done on real world
x2
performance impact at various error rates. Some theoretical 1
Then, using the same transmit swing settings, the than CPU Rx, such that at BER levels in the vicinity of 10-6,
performance of each of the three benchmarks was measured. performance penalties are similar.
Using this approach, performance vs. BER experienced at
the graphics card receiver could be characterized.
measured. The experiment was performed once with the supporting evidence that as BER is increased, the main area
timing sampling offset used to induce BER, and again with of concern for an end user is in fact performance degradation
the voltage sampling offset used to induce BER at the CPU rather than undetected data mismatch issues.
receiver.
C. PCIe Gen 3 link between CPU and PCH
In this experiment, a 2nd generation Intel Xeon E5
processor was connected to an Intel BD82C606 Server
Chipset Platform Control Hub (PCH) via a PCIe Gen 3
uplink. The intent was to study the impact of PCH Rx BER
on performance of the uplink.
Method used to Avg BER for 100% Avg BER for data
induce BER performance loss mismatch error
Timing sampling 3.0x10-7
Not measurable
offset (120 runs)
Voltage sampling 4.8x10-7 5.6x10-7 (measurable on
offset (120 runs) 8 of 120 runs)
D. SATA 6Gb/s link between PCH and hard disk drive experiments, performance began to show measurable
For this measurement, an Intel BD82C606 Server PCH decrease in the neighborhood of 10−8 or even 10−10.
SATA 6 Gb/s link attached to a hard drive was studied.
Similar to the experiment in the previous section, jitter
injection was used at the PCH Rx to induce a BER. Jitter
frequency and amplitude changes were made to vary the
BER, and for finer adjustments, temperature adjustments
were made in addition to a validation test hook that provided
some level of control of the PCH Rx voltage sampling point
with respect to the center of the data eye. As these
adjustments were being made, the BER could be calculated
by logging the disparity and CRC errors occurring on the
SATA link and dividing by the total number of bits
transmitted.
While errors were being induced in this manner, a
performance benchmark involving continuous reads and
writes to the hard drive was utilized to stress the SATA I/O
as well as monitor the performance at various levels of BER. Figure 5. PCH performance penalty vs. PCH Rx BER on the PCIe uplink
With the combination of jitter injection and the data eye to the CPU
margining hook, a reasonable level of accuracy was achieved
in inducing different levels of BER on the SATA link.
Similar to the PCH PCIe uplink experiment, errors were
induced and monitored while the performance monitor itself
was being run.
Fig. 6 shows the outcome of the experimental
measurements. As BER was increased above the spec of
10−12, minimal overall performance degradation was
witnessed until a BER level of approximately 3x10−10 was
achieved. At this level, a 3% performance penalty was
observed, but from that point on, the rate of performance
degradation with respect to BER increased dramatically. As
was so often witnessed in the experiments reported in this
whitepaper, 50% performance degradation occurred only at a
BER level one order of magnitude higher, at approximately
3x10−9. Figure 6. Performance loss vs. PCH Rx BER on the SATA 6Gb/s link to a
hard drive
IV. IMPLICATIONS AND FUTURE WORK This suggests that true effective latencies with real
The data presented here suggests for the high speed serial modern-day products and workloads, taking into account the
bus types studied, there are at least two orders of magnitude error profiles (for example, number of consecutive packets
of margin above the max BER specification before a user with errors), are sometimes greater than the assumed
would experience any noticable performance loss from penalties in [7]. While error ratio profiles could differ by
replaying data after an error is detected. It is worth scenario and would not always match those in the reported
mentioning, however, that the empirical data sometimes experiments, the fundamental sources of bit errors in the
showed lower margin than a simple latency-based theoretical experiments (jitter, elevated temperature, reduced transmit
projection would predict. Murali et al. [7] speculate that voltage swing, and a non-centered data sampling point) are
based on error retry-related latency penalties, average all sources that could be experienced in a real-world system.
observed latency would not show degradation until a packet To minimize the impact of extended test runs and using
or flit error ratio in the range of 0.1%-1%. In this context, more expensive design solutions to ensure parts meet the
latency refers to the amount of additional delay in the data BER spec, an alternative approach to a simple spec value
packet, or “flit,” that is created by the receiving device would be to architect in the right validation hooks and
notifying the transmitting device of the CRC error as well as capabilities to measure performance changes as data eye
the resend of the correct data by the transmitting device. margins decrease or alternatively, as BER increases.
Projecting this value onto PCIe Gen 3, for example, with a Validation activities can then concentrate on checking that
typical CRC-protected packet payload size of 1200-2200 bits the vast majority of parts and systems will not experience
for the products measured in this paper, one would predict noticeable performance penalties—for example, no more
there would be no performance concern until a BER elevates than 3% performance loss—from resending data across the
to the range of ~10−7 to 10−6. Yet in some of the link as a result of error detection. When needed, test content
such as the PCIe Gen 3 functional test used in this paper can types besides PCIe and SATA, as well as other scenarios for
be used to confirm that undetected data mismatch errors PCIe that include a graphics card AIC where hardware
happen at or above BER levels that create severe acceleration is turned off. It is also the hope that this work
performance degradation or hangs. will motivate others in the industry to perform studies on
By structuring validation targets with respect to their platform architectures.
performance, product validation teams can have confidence
they are truly validating for a quality end user experience, V. SUMMARY
rather than a generic BER level. To illustrate, the BER In this paper, four experiments were conducted to study
requirement of 10−12 is prevalent in specs for high speed the impact of increasing levels of BER on performance of
serial buses, despite different levels of error detection and high speed serial buses. On a PCIe Gen 3 link running
different retry time penalties on these various buses, not to between a CPU and a graphics add-in card, it was found that
mention product-level architectural differences that could although there were some card-to-card differences,
create different retry penalties product to product on the performance did not start to decrease from error-induced
same serial bus type. By forcing products to abide to one retries until a BER of 10−8 at the lowest. On a PCH PCIe
generic BER spec that is not explicitly tied to an end user Gen 3 uplink to a CPU as well as a SATA 6 Gb/s I/O
impact, the spec level must be overly conservative to account running from a PCH to a hard disk, performance did not
for all possible factors across all possible systems, implying appreciably decline until a BER of 10−10 or higher. Finally,
that most products are over-designing and over-validating. the PCIe Gen 3 functional test between a CPU and test add-
in card showed that catastrophic performance issues arose at
TABLE V. TEST TIME DIFFERENCES AT DIFFERENT BER LEVELS a BER of ~10−7 but that undetected mismatch errors do not
Min test time for 95% Min test time for 95% occur until the same level of BER or worse.
BER
requirement
confidence, PCIe G3 confidence, SATA 6 The data suggests that many products have additional
(seconds) Gb/s (seconds) margin above the 10−12 BER spec before any user impact
10-12 374 499 would occur. If new standards and practices were adopted to
10-10 3.74 4.99 validate against performance impact instead of a generic
BER specification level, conservatism leading to costly over-
In contrast, by aligning to a performance-based design and over-validation could be avoided.
requirement, this conservatism can be avoided, resulting in
additional design margin and shorter validation time. Design ACKNOWLEDGMENT
margin benefit is extremely difficult to quantify even on a The authors would like to thank Alejandro Cardenas,
single I/O type because of the enormous variety of Si circuit Federico Hernandez Reyes, David Steele, and Dale Robbins
designs, fabrication processes, and board designs. Validation for performance measurements on the PCH PCIe uplink and
time benefit is more straightforward to quantify, however. SATA. We would also like to thank Tsafrir Waller for the
To empirically confirm that a given link is less than or equal CPU/AIC PCIe performance test runs and analysis.
to a certain BER at a certain confidence level, one must test
for a sufficient time. The Poisson probability distribution REFERENCES
may be used to calculate the required length of test time to [1] PCI Express® Base 3.0 specification, www.pcisig.com
validate against a certain BER to a level of 95% confidence, [retrieved: Aug, 2014].
assuming no errors are encountered during the test: [2] IEEE 802.3TM-2012 Section 5, standards.ieee.org [retrieved:
Aug, 2014].
ln(1 0.95) . (2) [3] Serial ATA Revision 3.1 specification, www.sata-io.org
Min _ Test _ Time [retrieved: Aug, 2014].
BER dataRate
[4] Universal Serial Bus 3.1 Specification,
www.usb.org/developers/docs [retrieved: Aug, 2014].
Table V shows the test time improvement for PCIe Gen 3
and SATA 6 Gb/s using this approach. If an empirical [5] W. W. Peterson and D. T. Brown, “Cyclic codes for error
detection,” Proc. IRE, vol. 49, Jan. 1961, pp. 228-235.
validation test of this nature was implemented in a
[6] G. Castagnoli, S. Bräuer, and M. Herrmann,“Optimization of
manufacturing test, for example, this would imply a 99% cyclic redundancy-check codes with 24 and 32 parity bits,”
reduction in test time if it were confirmed that only a BER of IEEE Transactions on Communications, vol. 41, June 1993,
10−10 was needed as opposed to 10−12. This is immediately pp. 883-892.
evident from (2): test time is inversely proportional to BER. [7] S. Murali, T. Theocharides, N. Vijaykrishnan, M. J. Irwin, L.
One challenge encountered in this study was that, as far Benini, and G. De Micheli, “Analysis of error recovery
as the authors were able to discern, there is no published schemes for networks on chips,” IEEE Design and Test of
Computers, Sep-Oct 2005, pp. 435-442.
experimental data of performance penalties vs. BER on
[8] W. Liu and W. Lin, “Additive white gaussian noise level
modern high-speed interconnects to which a comparison estimation in SVD domain for images,” IEEE Transactions on
could be made. All previous investigations on this subject Image Processing, vol. 22, pp 872-88.
appear to be purely theoretical ([7][9]) and did not even [9] S. Wang, S. Sheu, H. Lee, T. O, “CPR: A CRC-Based Packet
analyze a specific existing high speed interconnect type. Recovery Mechanism for Wireless Networks,” 2013 IEEE
Because this appears to be an area not previously explored, Wireless Communications and Networking Conference, April
future studies will include other high speed interconnect 2013, pp. 321-326.