PCIe® 6.0 / 6.0.

1
Protocol Update
Paul Cassidy
PCI-SIG® Protocol Workgroup (PWG) Member
Synopsys
Disclaimer

• The information in this presentation refers to specifications still in the

development process. This presentation reflects the current thinking
of various PCI-SIG® workgroups, but all material is subject to
change before the specifications are released.

2
PCIe® 6.0 / 6.0.1 Protocol Update Agenda
Recent ECNs Against the 5.0/6.0 Base Spec
• Alternate Protocol DLLP Reservation
• TEE Device Interface Security Protocol (TDISP)
• Data Object Exchange (DOE) 1.1
Selected “ECRs” Under Development
• Unordered I/O (UIO)
• Address Translation Services (ATS) 1.2 / 2.0
• Sideband Signals Support for Internal Cable Spec
Major Protocol/Spec Changes for PCIe 6.0
• Flit Mode
• Segment Support in Flit Mode
• L0p replacing L0s in Flit Mode
• Shared Flow Control Credits
• “Strongly Recommended”  “Mandatory”
• 14-Bit Tags in Flit Mode
• Max_Payload_Size enhancements
• Deprecated features
• Improved SR-IOV spec integration
PCIe 6.0.1 Base Spec
• New Errata Delivery Mechanism
• Key Errata included in 6.0.1

3
Alternate Protocol DLLP Reservation ECN

4
Alternate Protocol Reservation ECN
• Make two DLLP encodings as available for
Alternate Protocol use
• Requested By CXL Consortium
• Receivers are required to silently ignore
unknown DLLPs
• The DLLPs are not expected on “regular”
PCIe Links, when an alternate protocol was
not negotiated

5
TEE Device Interface Security Protocol
(TDISP) ECN

6
TEE Device Interface Security Protocol (TDISP)
• Trusted execution environment (TEE) VMs
(TVMs) are used for host confidential
computing workloads that are isolated from
hosting environments where the VMM is
untrusted.
• When a portion of a device (i.e.; a TEE
Device Interface) such as a Virtual Function
when using SR-IOV, is assigned to a VM, it
is necessary to establish and maintain a
trusted execution environment.
• TDISP defines an architecture for trusted
I/O virtualization providing the following
functions:
1. Establishing a trust relationship between a TVM and a device.
2. Securing the interconnect between the host and device.
3. Attaching and detaching a TEE Device Interface to a TVM in a trusted manner.

7
Data Object Exchange (DOE) 1.1 ECN

8
Data Object Exchange (DOE), Revision 1.1
DOE Revision 1.1 defines or supports: Security Protocol and Data Model – SPDM
(DSP0274)
• Clarified DOE Discovery CMA/SPDM

• A Connection ID mechanism to support multiple SPDM over MCTP Binding

(DSP0275)
SPDM connections involving multiple software
entities operating through a single DOE instance Secured Messages using SPDM
(DSP0277)
• A mechanism for a DOE instance to notify the Secured Messages
host asynchronously that a device-initiated data using SPDM over
Secured
MCTP Binding SPDM
object is available (an “async” message vs. a SPDM
(DSP0276) Messages
Messages
response to a host-initiated request) over
over
MCTP Base DOE
• Support “alerting” in general, and specifically the DOE
(DSP0236) (data object
(data object
SPDM GET_ENCAPSULATED_REQUEST protocols
protocols
MCTP over SMBus MCTP over PCIe 01h & 03h)
mechanism 02h & 04h)
Binding Binding
(DSP0237) (DSP0238)
• An “attention” mechanism to support improved
power management of DOE implementations Legend: DMTF PCI-SIG

9
Unordered I/O (UIO) Draft ECN

10
Unordered IO (UIO) motivation
• Problems:
• Fabrics with multiple paths between a source and destination cannot be supported
• Posted Writes don’t match the semantics of other fabrics, in that the Requester doesn’t (directly) know
if/when a write actually completes
• Posted writes flowing towards destinations with differing write performance can cause global stalls
• Earlier efforts to relax the PCI/PCIe ordering rules such as Relaxed Ordering (RO) and ID-Based
Ordering (IDO) do not fully address the limitations noted above.
• UIO defines new wire semantics and related capabilities to address these problems
• Goals:
• Enable higher performance via source-ordering, including multipath fabrics
• Maintain full backwards compatibility with the PCI/PCIe producer-consumer model
• Provide the simplest possible discovery/configuration

11
Unordered IO (UIO) specifics
• Supported only with Flit Mode
• Relies on Flit Mode “earmarked” Type codes
• Thus, works with “existing” PCIe 6.0 Switches
• UIO traffic separated from non-UIO traffic
using existing TC/VC mechanisms
• Defines new/modified wire protocols
• Simplified (i.e., no) fabric ordering
• UIO Reads include variety of functional &
performance enhancements
• UIO Writes require Completions
• Multiple UIO Writes are permitted to use
same Tag value & Completions
• UIO Write Completions can be coalesced

12
Address Translation Services (ATS) 1.2 / 2.0

13
ATS 1.2 ECR
• Variety of errata plus minor ECR to clean up – tentatively:
• Items for removal/simplification/deprecation (things that aren’t really used and cause complexity)
• Remove G bit, EXE bit, Priv bit, U bit
• Remove allowance for multiple translations in one completion
• Remove allowance for multiple invalidations to be acknowledged in one completion
• Errata/bug-fixes/clarification
• Fix crediting for PASID Stop Requests
• Fix PRI crediting and auto-response issues
• Fix Invalidation flow control (especially for multi-Function devices)
• Remove “Tagged to be discarded”
• Fix/clean up "Implicit Invalidation"
• Fix/clean up consistency and proper spec language and form
• Consider minor additional capabilities to fix gaps:
• Translation request with write-only permissions requested
• Revise rules/requirements re No Snoop attribute
• Expect ATS 1.2 to complete in 2023

14
ATS 2.0

• Ongoing discussion to determine requirements, scope and timeline

• Foundational changes to ATS 1.x are potentially in-scope
• Considering one or more ECRs to define relatively major additional
capabilities, e.g.:
• PRI re-architecture
• Process info cache
• Anticipate that backwards compatibility will be modal – i.e., devices
operate in 1.x or 2.0 mode, host will distinguish new/old
• Expect ATS 2.0 completion in 2023

15
Sideband Signals Support for
Internal Cable Spec

16
Sideband ad-hoc Sub-team
• Developing / coordinating sideband development for in-progress form-factor work including CEM
6.0, new internal and external cable specs
• Will also create an ECR to the Base Spec for Architectural Sidebands:
• Add a chapter to the Base Specification to provide a “toolkit” of content related to sideband signals and
use – new & evolving form factor specifications can select elements from this chapter as appropriate to
fulfill their specific needs
• Include new elements, plus elements collected from existing form factor specifications
• Goals:
• By developing common elements across multiple form factors and use cases, the needed ingredients
will be more broadly developed and deployed.
• Form factor specifications will be easier to develop because they will not have to develop as much new
content but will instead reference this common toolkit.
• Rough anticipated outline (subject to change):
• Discrete sideband mechanisms (PERST, CLKREQ, etc.)
• Multiplexed mechanism for sideband scalability (tentatively called “PESTI”)
• Consistent addressing and use for “2-wire” busses (SMbus & I3C), including defined discovery
mechanisms, and as needed, control (e.g., for multiplexing of pins)

17
Major Protocol/Spec Changes for PCIe 6.0

• FLIT mode
• Segment Support in Flit Mode
• L0p replacing L0s in Flit Mode
• Shared Flow Control Credits
• “Strongly Recommended”  “Mandatory”
• 14-Bit Tags in Flit Mode
• Max_Payload_Size enhancements
• Deprecated features
• Improved SR-IOV spec integration

18
6.0 Changes:
Flit Mode Overview
• PCIe 6.0 Base Spec defines “Flit Mode” operation
• Required for data rates of 64.0 GT/s and future higher ones
• Discovered in hardware at Link initialization
• Once negotiated, applies to all data rates
• Totally new TLP Headers are used in Flit Mode
• Existing TLP Headers have many limitations; e.g., no room for increasing the Tag size
• Flit Mode TLP Headers parse more cleanly – existing TLP Headers are cumbersome
• TLP Translation occurs when forwarding between Flit Mode and Non-Flit Mode Links
• Exception: Selective IDE Stream packets cannot be translated
• Per-TLP Framing overhead becomes per-Flit overhead
• Small packets are now more efficient
• Large packets are now less efficient

19
6.0 Changes:
New TLP Header Formats for Flit Mode
• TLP Header is composed of a 3 to 7 DW TLP
Header Base, followed by 0 to 7 additional DWs of
“Orthogonal Header Content” (OHC)
First DW of FLIT Mode Header Base
• Single DW exceptions: NOP TLP and Reserved TLPs
• Fully-decoded 8b Packet Type field
• All 256 Type values are defined or earmarked to permit
proper framing and forwarding
• First DW of the Header Base includes all info required
to determine the full size of the TLP
64-bit Address Routed TLP – FLIT Mode • TLP Header Base, OHC, Payload, and Trailer
• Exception: Link-Local TLP Prefixes (rarely used)
• End-End TLP Prefixes are integrated into the Header
• Some are in architected OHC fields (e.g., PASID)
• Remaining ones become OHC-E
• Transaction Digest is replaced by a 0 to 5 DW “Trailer”
• Poison / nullify behavior is defined; two ways to poison
Example FLIT Mode OHC DWs

20
6.0 Changes:
Segment Support in Flit Mode
• Segment Numbers (a.k.a. Hierarchy IDs) identify unique
Configuration Address (BDF) spaces
• Root Complex support for multiple Segments dates to the 1990s,
but is becoming increasingly common
• Without Segment support in TLPs, tracking Non-Posted Requests
that route peer-to-peer across Segments requires the Root
Complex to “take ownership”, replacing the Transaction ID for the
Request and restoring it for the associated Completion(s)
• This increases implementation costs and/or reduces performance
for peer-to-peer traffic
• Lack of Segment information can also impede the analysis of
Advanced Error Reporting header logs
• For FLIT-Mode TLPs, when needed the Segment will be conveyed
in the header (e.g., the green path at right), avoiding the RC from
needing to take ownership
• Functions capture their Segment Number from Configuration Writes
• RCs should still remain capable of taking ownership when the FM: Flit Mode
path crosses a Non-FLIT-Mode Link, since this results in the loss NFM: Non-Flit Mode
of Segment information (e.g., red paths)

21
6.0 Changes:
L0p Replacing L0s for Flit Mode
• L0s is having problems!
• 8b N_FTS field is inadequate at higher speeds; is non-trivial to fix
• L0s does not work with retimers, which are increasingly used at higher
speeds
• L0s is less robust and less effective in power savings as speeds increase
• Marginal N_FTS is causing retraining failures with associated latency
penalties
• Basic overview of L0p ("L0 Partial")
• A substate of L0 where some Lanes are in a sleeping state
• Dynamic negotiated Link width without using Recovery
• Active Lanes continue operation while new Lanes train
• Can achieve power savings comparable to those in L1
• Link width changes at next SKP OS
• Retimers are required to support L0p
• L0p/L0s details
• L0p support: optional in Flit Mode; not supported in non-Flit Mode
• L0s support: not supported in Flit Mode; remains optional in non-Flit Mode

22
6.0 Changes:
Shared Flow Control Credits
• Multi-VC implementations need a full set of Rx buffers for each VC that needs to run at full speed
• This is getting increasingly expensive as data rates increase
• Shared Flow Control credits enables multiple VCs to share common pools of credits
• Flow control credit sharing is under control of the Receiver
• The Receiver indicates how it shares, and the remote Transmitter must comply
• The protocol is symmetric, though the sharing behavior can be different for each direction
• Negotiated using Data Link Feature Exchange
• Shared Flow Control support requires Scaled Flow Control support
• Extended VC Count is also negotiated – simplifies behavior if only 1 VC
• Flow Control Initialization
• VC0 still automatically initializes at Link Up
• Additional VCs still initialize when enabled by SW
• Six credit types exchanged (Dedicated P/NP/Cpl, Shared P/NP/Cpl)
• If supported and enabled, the additional VCs can contribute to the shared credit pools
• Dedicated VCn credits ensure forward progress
• Must cover the max size TLP of the appropriate flow control class
• Completions optionally share Posted Request shared credit pools
• Software should set the new “I’m done initializing VCs” config bit
• Tells hardware that no additional VCs will be enabled so it can distribute unused buffers across existing VCs

23
6.0 Changes:
“Strongly Recommended”  “Mandatory”
• In previous spec versions, “strongly recommended” was sometimes used in place of “must” to
prevent older implementations from becoming non-compliant
• Flit Mode recognized as requiring a major silicon redesign, and thus providing a rare opportunity to
improve implementation robustness without breaking spec backward compatibility with “old silicon”
• For 6.0 spec, reviewed all occurrences of “strongly recommended” and “highly recommended”
• New term “MUST@FLIT” added; meaning:
• “Must” for devices that support Flit Mode
• “Strongly recommended” (if applicable) for devices that do not support Flit Mode
• Strongly recommended defined as not required but urged to motivate robust implementations
• “Highly recommended” phrase is being retired in favor of “strongly recommended”
• Examples:
• Was: It is strongly recommended that the Completion Timeout mechanism not expire in less than 10 ms.
• Now: The Completion Timeout mechanism MUST@FLIT not expire in less than 10 ms.
• Was: It is strongly recommended that this bit be Set in all PTM devices.
• Now: This bit MUST@FLIT be Set in all PTM Devices.

24
6.0 Changes:
14-bit Tags in Flit Mode
• Applicable TLP Headers in Flit Mode carry 14-bit Tags
• 14-bit Tag support adds yet another size to PCIe’s existing set of 10-bit, 8-bit, and 5-bit Tags
• With this increase in size, the expectation is that 14-bit Tags will serve PCIe well “for many years to come”
• 14-bit Tag support adds a set of Capability & Control bits analogous to those for 10-bit Tags
• 14-bit Tag Completer Supported – this bit MUST@FLIT be Set
• 14-Bit Tag Requester Supported & 14-Bit Tag Requester Enable
• VF 14-Bit Tag Requester Supported & VF 14-Bit Tag Requester Enable
• 14-bit Tag support builds upon key paradigm elements established by 10-bit Tags
• Restricted 14-bit Tag ranges for “valid” Tags enable error detection by the Requester if the Completer or path
routing elements return invalid upper Tag bits (usually due to them not supporting 14-bit Tags).
• For a given Endpoint (EP), System Software is encouraged to enable the greatest common Requester Tag
size within its Tag capability, the host’s DMA Tag capability, and the EP’s path to the host.
• If the EP supports P2P with other EPs, the EP’s driver must make any adjustments to its enabled Requester
Tag size as necessary to accommodate the peer EP’s Tag capability as well as the P2P path.
• If an EP supports an implementation-specific mech for sending different Tag-size Requests to different
Completers, the EP driver must configure that mech to accommodate any associated Tag size capabilities.

25
6.0 Changes:
Max_Payload_Size (MPS) enhancements
• For the 0.9 draft of the 6.0 Base spec, there are three major MPS enhancements
• These were “late additions” designed to take advantage of the rare MUST@FLIT opportunity
• MPS Capability MUST@FLIT support a minimum of 512 bytes
• This is a major increase over the architected min of 128 bytes, but is reasonable for Flit Mode implementations
• Modifying MPS settings without quiescing traffic
• A new Rx_MPS_Fixed capability (MUST@FLIT) decouples the Rx MPS limit from the Tx MPS setting
• This enables SW to change the Tx MPS setting without needing to quiesce traffic
• This avoids major hot-plug limitations; e.g., an AIC hot-add being denied due to its small MPS capability
• Utilizing different MPS settings for different targets
• A new Mixed_MPS_Supported capability bit indicates supporting different MPS settings for different targets
• MUST@FLIT be Set if Function supports P2P and its MPS capability is >512 bytes
• Enables clusters of EPs using large MPS TLPs (e.g., 2048 bytes) for P2P even if the RC’s MPS is much smaller
• For case where host mem supports Rx_MPS_Fixed, System Software is encouraged to base each EP’s MPS
setting upon host mem MPS capability and the EP’s path to host mem.
• If the EP supports P2P with other EPs, the EP’s driver must make any adjustments to its MPS settings as
necessary to accommodate the peer EP’s MPS capabilities as well as the P2P path.

26
6.0 Changes:
Deprecated Features
• Features deprecated:
• Multi-Root I/O Virtualization (MR-IOV)
• Lightweight Notification (LN) Protocol
• x32 and x12 Link Widths

• Features not supported in Flit Mode, but with continued support in Non-Flit Mode:
• ASPM (Active State Power Management) L0s
• Protocol Multiplexing (PMUX)

• Features earlier considered for deprecation, but will continue to be supported:

• ID-Based Ordering (IDO)
• INTx Mechanism and Level-Sensitive Interrupts
• I/O Space and I/O Read/Write Transactions
• Byte Enables in TLP Headers

27
6.0 Changes:
Improved SR-IOV spec integration
• The Single Root I/O Virtualization & Sharing (SR-IOV) spec was originally distinct from the Base
• The 4.0 Base spec integrated SR-IOV material, placing most of it in Chapter 9 with few updates
• Due to time constraints, little Ch9 material was merged elsewhere, which caused problems
• Ch9 placed many requirements on registers & fields defined in Ch7, but some Ch9 material was stale
• Several registers & fields subsequently added in Ch7 didn’t get addressed properly in Ch9
• A major effort improving SR-IOV (Ch9) integration accomplished the following:
• Merged all material covering Ch7 register changes into Ch7
• Merged all material covering ATS changes into Ch10
• Merged all SR-IOV Error Handling material elsewhere in the spec
• Merged all SR-IOV Power Management material into Ch5
• Removed essentially all material related to MR-IOV as part of the effort to deprecate MR-IOV
• End result
• Ch9 in 5.0 Base spec is 70 pages; Ch9 in 6.0 Base spec is 36 pages
• Several inconsistencies between Ch9 & the rest of the Base spec were resolved
• SR-IOV content in the Base spec will be significantly easier to understand & maintain
28
PCIe 6.0.1 Base Spec

29
New Errata Delivery Mechanism
• Errata now integrated into the Base spec
• No standalone “change this to that” errata document
• Other specs continue current practice(s)
• 6.0 is a new revision of Base spec (new data rate)
• 6.1 is 6.0 + approved ECNs (staple gun document – nothing new other than ECNs / Errata)
• 6.n.m is Errata document m relative to 6.n (e.g., 6.0.1)
• Changebar and no changebar versions
• Errata Table follows Revision History

• Navigation Hints:
• Search for ⇅ to locate all changes
• There’s a ⇅ character on page 1 to make this easier
• Errata are “chained” together
• △ Takes you to the Errata Table entry
• ◁ Takes you to the previous change for this errata
• ▷ Takes you to the next change for this errata
• Inserted Text: ↑green text with yellow highlighting↑
• Deleted Text: ↓red text with red highlighting↓

30
Key Errata included in PCIe 6.0.1
• A20 Partial Header Encryption corrections:
• Encryption must start on byte boundaries, not bit boundaries
• Define Partial Header Encryption behavior for Link IDE
• A27 Clarify Flit Mode Receiver parsing requirements regarding of the Length field
• A35 Shared Flow Control Corrections:
• Define behavior for VCs that are enabled, disabled, and re-enabled
• Clarify rules in Table 3-2
• Correct encoding of Symbol +0 bit 3 in § Figure 3-9, Figure 3-10, and Figure 3-11 to match Table 3-2
• Shared Flow Control Usage Limit is RsvdP for Non-Flit Mode components.
• A38 L0p Corrections:
• Preserve LFSR relationship between Lanes on width increase
• Update scrambler behavior during L0p width increase
• Update Link Management DLLP Behavior
• A53 Add Flit Error Counter Interrupt Controls
• A62 Clarify Flit Mode TLP Reserved bit / encoding behavior at the Receiver
• A67 Training Set Corrections:
• Clarify equalization fields in TS0 Ordered Sets (see Table 4-29)

31