Intel® FPGA P-Tile Avalon®

Streaming IP for PCI Express* User

Guide
Updated for Intel® Quartus® Prime Design Suite: 20.4

IP Version: 4.0.0

Subscribe UG-20225 | 2021.02.18

Send Feedback Latest document on the web: PDF | HTML
 Contents

Contents

1. Introduction................................................................................................................... 5
1.1. Overview..............................................................................................................5
1.2. Features...............................................................................................................5
1.3. Release Information...............................................................................................8
1.4. Device Family Support............................................................................................9
1.5. Performance and Resource Utilization....................................................................... 9
1.6. IP Core and Design Example Support Levels............................................................ 10
2. IP Architecture and Functional Description................................................................... 12
2.1. Architecture........................................................................................................ 12
2.1.1. Clock Domains.........................................................................................13
2.1.2. Refclk.....................................................................................................14
2.1.3. Reset..................................................................................................... 16
2.2. Functional Description.......................................................................................... 17
2.2.1. PMA/PCS................................................................................................ 17
2.2.2. Data Link Layer Overview..........................................................................18
2.2.3. Transaction Layer Overview....................................................................... 20
3. Parameters................................................................................................................... 22
3.1. Top-Level Settings............................................................................................... 22
3.2. Core Parameters.................................................................................................. 24
3.2.1. System Parameters.................................................................................. 27
3.2.2. Avalon Parameters................................................................................... 27
3.2.3. Base Address Registers.............................................................................28
3.2.4. Multi-function and SR-IOV......................................................................... 29
3.2.5. TLP Processing Hints (TPH)/Address Translation Services (ATS) Capabilities..... 35
3.2.6. PCI Express and PCI Capabilities Parameters............................................... 35
3.2.7. Device Identification Registers................................................................... 42
3.2.8. Configuration, Debug and Extension Options................................................43
4. Interfaces..................................................................................................................... 45
4.1. Overview............................................................................................................ 45
4.2. Clocks and Resets................................................................................................ 48
4.2.1. Interface Clock Signals............................................................................. 48
4.2.2. Resets.................................................................................................... 49
4.3. Serial Data Interface............................................................................................ 51
4.4. Avalon-ST Interface ............................................................................................ 51
4.4.1. TLP Header and Data Alignment for the Avalon-ST RX and TX Interfaces......... 52
4.4.2. Avalon-ST RX Interface.............................................................................52
4.4.3. Avalon-ST RX Interface rx_st_ready Behavior...........................................58
4.4.4. RX Flow Control Interface..........................................................................59
4.4.5. Avalon-ST TX Interface ............................................................................ 61
4.4.6. Avalon-ST TX Interface tx_st_ready Behavior...........................................67
4.4.7. TX Flow Control Interface..........................................................................68
4.4.8. Tag Allocation.......................................................................................... 69
4.5. Hard IP Status Interface....................................................................................... 70
4.6. Interrupt Interface............................................................................................... 71
4.6.1. Legacy Interrupts.................................................................................... 72

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

2
Contents

4.6.2. MSI........................................................................................................73
4.6.3. MSI-X.....................................................................................................76
4.7. Error Interface.....................................................................................................79
4.7.1. Completion Timeout Interface....................................................................81
4.8. Hot Plug Interface (RP Only)..................................................................................84
4.9. Power Management Interface................................................................................ 85
4.10. Configuration Output Interface.............................................................................87
4.11. Configuration Intercept Interface (EP Only)........................................................... 91
4.12. Hard IP Reconfiguration Interface......................................................................... 93
4.12.1. Address Map for the User Avalon-MM Interface...........................................95
4.12.2. Configuration Registers Access................................................................. 98
4.13. PHY Reconfiguration Interface............................................................................ 101
4.14. Page Request Service (PRS) Interface (EP Only)................................................... 102
5. Advanced Features......................................................................................................104
5.1. PCIe Port Bifurcation and PHY Channel Mapping..................................................... 104
5.2. Virtualization Support......................................................................................... 104
5.2.1. SR-IOV Support..................................................................................... 104
5.2.2. VirtIO Support....................................................................................... 110
5.3. TLP Bypass Mode............................................................................................... 121
5.3.1. Overview.............................................................................................. 122
5.3.2. Register Settings for the TLP Bypass Mode.................................................122
5.3.3. User Avalon-MM Interface....................................................................... 124
5.3.4. Avalon-ST Interface ...............................................................................125
6. Testbench................................................................................................................... 127
6.1. Endpoint Testbench............................................................................................ 128
6.2. Test Driver Module............................................................................................. 129
6.3. Root Port BFM....................................................................................................130
6.3.1. BFM Memory Map...................................................................................131
6.3.2. Configuration Space Bus and Device Numbering......................................... 132
6.3.3. Configuration of Root Port and Endpoint.................................................... 132
6.3.4. Issuing Read and Write Transactions to the Application Layer....................... 138
6.4. BFM Procedures and Functions ............................................................................ 138
6.4.1. ebfm_barwr Procedure ...........................................................................138
6.4.2. ebfm_barwr_imm Procedure ................................................................... 139
6.4.3. ebfm_barrd_wait Procedure .................................................................... 139
6.4.4. ebfm_barrd_nowt Procedure ...................................................................140
6.4.5. ebfm_cfgwr_imm_wait Procedure ............................................................ 140
6.4.6. ebfm_cfgwr_imm_nowt Procedure ........................................................... 141
6.4.7. ebfm_cfgrd_wait Procedure .................................................................... 141
6.4.8. ebfm_cfgrd_nowt Procedure ................................................................... 142
6.4.9. BFM Configuration Procedures..................................................................142
6.4.10. BFM Shared Memory Access Procedures ................................................. 144
6.4.11. BFM Log and Message Procedures ..........................................................146
6.4.12. Verilog HDL Formatting Functions .......................................................... 149
7. Troubleshooting/Debugging....................................................................................... 153
7.1. Hardware.......................................................................................................... 153
7.1.1. Debugging Link Training Issues................................................................ 154
7.1.2. Debugging Data Transfer and Performance Issues.......................................160
7.2. Debug Toolkit.................................................................................................... 164

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

3
 Contents

7.2.1. Overview.............................................................................................. 164

7.2.2. Enabling the P-Tile Debug Toolkit..............................................................166
7.2.3. Launching the P-Tile Debug Toolkit........................................................... 166
7.2.4. Using the P-Tile Debug Toolkit.................................................................. 168
7.2.5. Enabling the P-Tile Link Inspector............................................................. 178
7.2.6. Using the P-Tile Link Inspector................................................................. 179
8. Document Revision History......................................................................................... 182
8.1. Document Revision History for the Intel FPGA P-Tile Avalon Streaming IP for PCI
Express User Guide..........................................................................................182
A. Configuration Space Registers.................................................................................... 184
A.1. Configuration Space Registers..............................................................................184
A.1.1. Register Access Definitions...................................................................... 186
A.1.2. PCIe Configuration Header Registers.........................................................186
A.1.3. PCI Express Capability Structures.............................................................188
A.1.4. Physical Layer 16.0 GT/s Extended Capability Structure...............................190
A.1.5. MSI-X Registers..................................................................................... 190
A.2. Configuration Space Registers for Virtualization......................................................192
A.2.1. SR-IOV Virtualization Extended Capabilities Registers Address Map............... 192
A.2.2. PCIe Configuration Registers for Each Virtual Function.................................192
A.3. Intel-Defined VSEC Capability Registers................................................................ 198
A.3.1. Intel-Defined VSEC Capability Header (Offset 00h)..................................... 198
A.3.2. Intel-Defined Vendor Specific Header (Offset 04h)...................................... 199
A.3.3. Intel Marker (Offset 08h)........................................................................ 199
A.3.4. JTAG Silicon ID (Offset 0x0C - 0x18)........................................................ 199
A.3.5. User Configurable Device and Board ID (Offset 0x1C - 0x1D)....................... 200
A.3.6. General Purpose Control and Status Register (Offset 0x30)..........................200
A.3.7. Uncorrectable Internal Error Status Register (Offset 0x34)...........................201
A.3.8. Uncorrectable Internal Error Mask Register (Offset 0x38)............................ 201
A.3.9. Correctable Internal Error Status Register (Offset 0x3C)..............................202
A.3.10. Correctable Internal Error Mask Register (Offset 0x40).............................. 203
B. Implementation of Address Translation Services (ATS) in Endpoint Mode.................. 204
B.1. Sending Translated/Untranslated Requests............................................................ 204
B.2. Sending a Page Request Message from the Endpoint (EP) to the Root Complex (RC)... 204
B.3. Invalidating Requests/Completions....................................................................... 204
C. Packets Forwarded to the User Application in TLP Bypass Mode................................. 205
C.1. EP TLP Bypass Mode (Upstream).......................................................................... 205
C.2. RC TLP Bypass Mode (Downstream)..................................................................... 210
D. Using the Avery BFM for Intel P-Tile PCI Express Gen4 x16 Simulations.................... 216
D.1. Overview.......................................................................................................... 216
D.2. Generate the PCIe PIO Example Design................................................................ 217
D.2.1. Avalon-ST PCIe PIO Example Design.........................................................217
D.3. Integrate Avery BFMs (Root Complex).................................................................. 217
D.4. Configure the Avery BFM and Update the Simulation Script......................................218
D.5. Compile and Simulate.........................................................................................219
D.6. View the Results................................................................................................ 220

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

4
 UG-20225 | 2021.02.18

Send Feedback

1. Introduction

1.1. Overview
P-Tile is an FPGA companion tile die that supports PCI Express* Gen4 in Endpoint,
Root Port and TLP Bypass modes.

It serves as a companion tile for both Intel® Stratix® 10 DX and Intel Agilex™ devices.

P-Tile natively supports PCI Express Gen3 and Gen4 configurations.

Related Information
Intel FPGA P-Tile Avalon streaming IP for PCI Express Design Example User Guide

1.2. Features
The P-tile Avalon® streaming IP for PCI Express supports the following features:

PCIe* Features:
• Complete protocol stack including the Transaction, Data Link, and Physical Layers
implemented as a Hard IP.
• Configurations supported:

Table 1. Configurations Supported by the P-Tile Avalon streaming IP for PCI Express
Gen3/Gen4 x16 Gen3/Gen4 x8 Gen3/Gen4 x4

Endpoint (EP) Yes Yes N/A

Root Port (RP) Yes N/A Yes

TLP Bypass Yes Yes Yes

Note: Gen1/Gen2 configurations are supported via link down-training.

• Static port bifurcation (four x4s Root Port, two x8s Endpoint).
• Supports TLP Bypass mode.
— Supports one x16, two x8, or four x4 interfaces.
— Supports upstream/downstream TLP bypass mode.
• Supports up to 512-byte maximum payload size (MPS).
• Supports up to 4096-byte (4 KB) maximum read request size (MRRS).
• Single Virtual Channel (VC).
• Page Request Services (PRS).
• Completion Timeout Ranges.

Intel Corporation. All rights reserved. Agilex, Altera, Arria, Cyclone, Enpirion, Intel, the Intel logo, MAX, Nios,
Quartus and Stratix words and logos are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or
other countries. Intel warrants performance of its FPGA and semiconductor products to current specifications in ISO
accordance with Intel's standard warranty, but reserves the right to make changes to any products and services 9001:2015
at any time without notice. Intel assumes no responsibility or liability arising out of the application or use of any Registered
information, product, or service described herein except as expressly agreed to in writing by Intel. Intel
customers are advised to obtain the latest version of device specifications before relying on any published
information and before placing orders for products or services.
*Other names and brands may be claimed as the property of others.
 1. Introduction
UG-20225 | 2021.02.18

• Atomic Operations (FetchAdd/Swap/CAS).

• Extended Tag Support.
— 10-bit Tag Support (Port 0 x16 Controller only)
• Separate Refclk with Independent Spread Spectrum Clocking (SRIS).
• Separate Refclk with no Spread Spectrum Clocking (SRNS).
• Common Refclk architecture.
• PCI Express Advanced Error Reporting (PF only).
Note: Advanced Error Reporting is always enabled in the P-Tile Avalon streaming
IP for PCIe.
• ECRC generation and checking.
• Data bus parity protection.
• Supports D0 and D3 PCIe power states.
• Lane Margining at Receiver.
• Retimers presence detection.

Multifunction and Virtualization Features:

• SR-IOV support (8 PFs, 2K VFs per each Endpoint).
• Access Control Service (ACS) capability.
Note: For ACS, only ports 0 and 1 are supported.
• Alternative Routing-ID Interpretation (ARI).
• Function Level Reset (FLR).
• TLP Processing Hint (TPH).
Note: TPH supports the "No Steering Tag (ST)" mode only.
• Address Translation Services (ATS). (For more information, refer to
Implementation of Address Translation Services (ATS) in Endpoint Mode on page
204).
• Process Address Space ID (PasID).
• Configuration Intercept Interface (for VirtIO).

IP Features:

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

6
1. Introduction
UG-20225 | 2021.02.18

• User packet interface with separate header, data and prefix.

• User packet interface with a split-bus architecture where the header, data and
prefix busses consist of two segments each (x16 mode only). This improves the
bandwidth efficiency of this interface as it can handle up to 2 TLPs in any given
cycle.
• Maximum numbers of outstanding Non-Posted requests (NPRs) supported when 8-
bit tags or 10-bit tags are enabled are summarized in the table below:

Table 2. Outstanding Non-Posted Requests Supported

Ports Active Cores 8-bit Tags 10-bit Tags

0 x16 256 512 (*)

1 x8 256 N/A

2 and 3 x4 256 N/A

Note: (*): Use tags 256 to 767.

• Completion timeout interface.
— The PCIe Hard IP can optionally track outgoing non-posted packets to report
completion timeout information to the application.
• You cannot change the pin allocations for the P-Tile Avalon streaming IP for PCI
Express in the Intel Quartus® Prime project. However, this IP does support lane
reversal and polarity inversion on the PCB by default.
• Supports Autonomous Hard IP mode.
— This mode allows the PCIe Hard IP to communicate with the Host before the
FPGA configuration and entry into User mode are complete.
Note: Unless Readiness Notifications mechanisms are used, the Root Complex
and/or system software must allow at least 1.0 s after a Conventional
Reset of a device before it may determine that a device that fails to
return a Successful Completion status for a valid Configuration Request
is a broken device. This period is independent of how quickly Link
training completes.
• FPGA core configuration via PCIe link (CvP Init and CvP Update).
Note: CvP Init and CvP Update are available for Intel Stratix 10 DX devices. For
Intel Agilex devices, CvP Init is available, and CvP Update will be available in
a future Intel Quartus Prime release.
Note: For Gen3 and Gen4 x16 variants, Port 0 (corresponding to lanes 0 - 15)
supports the CvP features. For Gen3 and Gen4 x8 variants, only Port 0
(corresponding to lanes 0 - 7) supports the CvP features. Port 1
(corresponding to lanes 8 - 15) does not support CvP.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

7
 1. Introduction
UG-20225 | 2021.02.18

• Device-dependent PLD clock (coreclkout_hip) frequency.

— 350 MHz / 400 MHz for Intel Stratix 10 DX devices, 350 MHz / 400 MHz / 500
MHz for Intel Agilex devices.
• P-Tile Debug Toolkit including the following features:
— Protocol and link status information.
— Basic and advanced debugging capabilities including PMA register access and
Eye viewing capability.
• Modelsim and VCS are the simulators supported in the 20.3 release of Intel
Quartus Prime. Other simulators may be supported in a future release.

Note: Throughout this User Guide, the term Avalon-ST may be used as an abbreviation for
the Avalon streaming interface or IP.

1.3. Release Information

Table 3. P-Tile Avalon streaming IP for PCI Express Release Information
Item Description

IP Version 4.0.0

Intel Quartus Prime Version 20.4

Release Date December 2020

Ordering Codes No ordering code is required

IP versions are the same as the Intel Quartus Prime Design Suite software versions up
to v19.1. From Intel Quartus Prime Design Suite software version 19.2 or later, IPs
have a new IP versioning scheme.

The IP version (X.Y.Z) number may change from one Intel Quartus Prime software
version to another. A change in:
• X indicates a major revision of the IP. If you update your Intel Quartus Prime
software, you must regenerate the IP.
• Y indicates the IP includes new features. Regenerate your IP to include these new
features.
• Z indicates the IP includes minor changes. Regenerate your IP to include these
changes.

Intel verifies that the current version of the Intel Quartus Prime Pro Edition software
compiles the previous version of each IP core, if this IP core was included in the
previous release. Intel reports any exceptions to this verification in the Intel IP
Release Notes or clarifies them in the Intel Quartus Prime Pro Edition IP Update tool.
Intel does not verify compilation with IP core versions older than the previous release.

Related Information
P-Tile IP for PCI Express IP Core Release Notes

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

8
1. Introduction
UG-20225 | 2021.02.18

1.4. Device Family Support

The following terms define device support levels for Intel FPGA IP cores:
• Advance support—the IP core is available for simulation and compilation for this
device family. Timing models include initial engineering estimates of delays based
on early post-layout information. The timing models are subject to change as
silicon testing improves the correlation between the actual silicon and the timing
models. You can use this IP core for system architecture and resource utilization
studies, simulation, pinout, system latency assessments, basic timing assessments
(pipeline budgeting), and I/O transfer strategy (data-path width, burst depth, I/O
standards tradeoffs).
• Preliminary support—the IP core is verified with preliminary timing models for
this device family. The IP core meets all functional requirements, but might still be
undergoing timing analysis for the device family. It can be used in production
designs with caution.
• Final support—the IP core is verified with final timing models for this device
family. The IP core meets all functional and timing requirements for the device
family and can be used in production designs.

Table 4. Device Family Support

Device Family Support Level

Intel Stratix 10 DX Final support

Intel Agilex Preliminary support

No support
Other device families Refer to the Intel PCI Express Solutions web page on the Intel website for support information on
other device families.

1.5. Performance and Resource Utilization

The following table shows the recommended FPGA fabric speed grades for all the
configurations that the Avalon-ST IP core supports.

Table 5. Intel Stratix 10 DX / Intel Agilex Recommended FPGA Fabric Speed Grades
for All Avalon-ST Widths and Frequencies
The recommended FPGA fabric speed grades are for production parts.

Lane Rate Link Width Application Interface Application Clock Recommended FPGA
Data Width Frequency (MHz) Fabric Speed Grades

350 MHz / 400 MHz

(Intel Stratix 10 DX)
x4 128-bit -1, -2
350 MHz / 400 MHz /
500 MHz (Intel Agilex)

350 MHz / 400 MHz

(Intel Stratix 10 DX)
Gen4 x8 256-bit -1, -2
350 MHz / 400 MHz /
500 MHz (Intel Agilex)

350 MHz / 400 MHz

(Intel Stratix 10 DX)
x16 512-bit -1, -2
350 MHz / 400 MHz /
500 MHz (Intel Agilex)
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

9
 1. Introduction
UG-20225 | 2021.02.18

Lane Rate Link Width Application Interface Application Clock Recommended FPGA
Data Width Frequency (MHz) Fabric Speed Grades

x4 128-bit 250 MHz -1, -2, -3

Gen3 x8 256-bit 250 MHz -1, -2, -3

x16 512-bit 250 MHz -1, -2, -3

The following table shows the typical resource utilization information for selected
configurations.

The resource usage is based on the Avalon-ST IP core top-level entity

(intel_pcie_ptile_ast) that includes IP core soft logic implemented in the FPGA
fabric.

Table 6. Resource Utilization Information for the PIO Design Example

Design Example Link Device Family ALMs M20Ks Logic Registers
Used Configuration

Programmed I/O Gen4 x16, EP Intel Stratix 10 3,191 0 10,255

(PIO) DX

Programmed I/O Gen4 x16, EP Intel Agilex 3,513 0 9,896

(PIO)

For details on the application clock frequencies that the IP core can support, refer to
Table 10 on page 14.

1.6. IP Core and Design Example Support Levels

The following table shows the support levels of the Avalon-ST IP core and design
example in Intel Stratix 10 DX devices.

Table 7. P-Tile Avalon Streaming (Avalon-ST) IP for PCIe Support Matrix for Intel
Stratix 10 DX Devices
Support level keys: S = simulation, C = compilation, T = timing, H = hardware, N/A = configuration not
supported

PCIe IP Support Design Example Support

Configuration
EP RP BP EP RP BP

Gen4 x16 512-

SCTH SCTH SCTH SCTH N/A N/A
bit

Gen4 x8/x8
SCTH N/A SCTH SCTH N/A N/A
256-bit

Gen4
x4/x4/x4/x4 N/A SCTH SCTH N/A N/A N/A
128-bit

Gen3 x16 512-

SCTH SCTH SCTH SCTH N/A N/A
bit

Gen3 x8/x8
SCTH N/A SCTH SCTH N/A N/A
256-bit

Gen3
x4/x4/x4/x4 N/A SCTH SCTH N/A N/A N/A
128-bit

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

10
1. Introduction
UG-20225 | 2021.02.18

The following table shows the support levels of the Avalon-ST IP core and design
example in Intel Agilex devices.

Table 8. P-Tile Avalon Streaming (Avalon-ST) IP for PCIe Support Matrix for Intel
Agilex Devices
Support level keys: S = simulation, C = compilation, T = timing, H = hardware, N/A = configuration not
supported

PCIe IP Support Design Example Support

Configuration
EP RP BP EP RP BP

Gen4 x16 512-

SCTH SCTH SCTH SCTH N/A N/A
bit

Gen4 x8/x8
SCTH N/A SCTH SCTH N/A N/A
256-bit

Gen4
x4/x4/x4/x4 N/A SCTH SCTH N/A N/A N/A
128-bit

Gen3 x16 512-

SCTH SCTH SCTH SCTH N/A N/A
bit

Gen3 x8/x8
SCTH N/A SCTH SCTH N/A N/A
256-bit

Gen3
x4/x4/x4/x4 N/A SCTH SCTH N/A N/A N/A
128-bit

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

11
 UG-20225 | 2021.02.18

Send Feedback

2. IP Architecture and Functional Description

2.1. Architecture
The P-tile Avalon-ST IP for PCI Express consists of the following major sub-blocks:
• PMA/PCS
• Four PCIe cores (one x16 core, one x8 core and two x4 cores)
• Embedded Multi-die Interconnect Bridge (EMIB)
• Soft logic blocks in the FPGA fabric to implement functions such as VirtIO, etc.

Figure 1. P-tile Avalon-ST IP for PCI Express top-level block diagram

P-Tile Avalon-ST PCIe IP Top Level

P-Tile EMIB FPGA Fabric

PHY
PMA Quad 3 PCIe Controllers
PLL A/B x4
PHY Datax4 Trans-
PMA Quad 2 Layer
PHY Link Datax8action
Trans-
Bifurcation Mux

PLL A/B (MAC)

LayerPHYLayer Layer
LinkData action IP
PCIe x16 Trans-
PCIe (MAC)
Layer LayerLinkx16Layer
action
Core User
x16 Lanes PMA Quad 1 PCS PHY Data Trans- Soft Logic
(MAC)
Layer Layer Link Layeraction Logic
PLL A/B (MAC) Layer Layer
refclk0
refclk1 PMA Quad 0
PLL A/B
pin_perst_n

Note: Each core in the PCIe Hard IP implements its own Data Link Layer and Transaction
Layer.

Intel Corporation. All rights reserved. Agilex, Altera, Arria, Cyclone, Enpirion, Intel, the Intel logo, MAX, Nios,
Quartus and Stratix words and logos are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or
other countries. Intel warrants performance of its FPGA and semiconductor products to current specifications in ISO
accordance with Intel's standard warranty, but reserves the right to make changes to any products and services 9001:2015
at any time without notice. Intel assumes no responsibility or liability arising out of the application or use of any Registered
information, product, or service described herein except as expressly agreed to in writing by Intel. Intel
customers are advised to obtain the latest version of device specifications before relying on any published
information and before placing orders for products or services.
*Other names and brands may be claimed as the property of others.
2. IP Architecture and Functional Description
UG-20225 | 2021.02.18

The four cores in the PCIe Hard IP can be configured to support the following
topologies:

Table 9. Configuration Modes Supported by the P-tile Avalon-ST IP for PCI Express
Endpoint
(EP) / Root
Configuration Mode Native IP Mode Port (RP) / Active Cores
TLP Bypass
(BP)

Configuration Mode 0 Gen3x16 or Gen4x16 EP/RP/BP x16

Configuration Mode 1 Gen3x8/Gen3x8 or Gen4x8/Gen4x8 EP/BP x16, x8

Gen3x4/Gen3x4/Gen3x4/Gen3x4 or
Configuration Mode 2 RP/BP x16, x8, x4_0, x4_1
Gen4x4/Gen4x4/Gen4x4/Gen4x4

In Configuration Mode 0, only the x16 core is active, and it operates in x16 mode (in
either Gen3 or Gen4).

In Configuration Mode 1, the x16 core and x8 core are active, and they operate as two
Gen3 x8 cores or two Gen4 x8 cores.

Note: When you use only one of the x8 bifurcated ports, you must ensure that the other
bifurcated port's lanes are not physically connected.

In Configuration Mode 2, all four cores (x16, x8, x4_0, x4_1) are active, and they
operate as four Gen3 x4 cores or four Gen4 x4 cores.

Each of the cores has its own Avalon-ST interface to the user logic. The number of IP-
to-User Logic interfaces exposed to the FPGA fabric are different based on the
configuration modes. For more details, refer to the Overview section of the Interfaces
chapter.

2.1.1. Clock Domains

The P-Tile IP for PCI Express has three primary clock domains:
• PHY clock domain (i.e. core_clk domain): this clock is synchronous to the
SerDes parallel clock.
• EMIB/FPGA fabric interface clock domain (i.e. pld_clk domain): this clock is
derived from the same reference clock (refclk0) as the one used by the SerDes.
However, this clock is generated from a stand-alone core PLL.
• Application clock domain (coreclkout_hip): this clock is an output from the P-
Tile IP, and it has the same frequency as pld_clk.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

13
 2. IP Architecture and Functional Description
UG-20225 | 2021.02.18

Figure 2. Clock Domains

FPGA Fabric P-Tile x16 core_clk

x8 core_clk
x4_0 core_clk
x4_1 core_clk
P P
User P-Tile
EMIB PCIe Hard IP C M pld_clk
Logic Wrapper
S A
Logic coreclkout_hip

The PHY clock domain (i.e. core_clk domain) is a dynamic frequency domain. The
PHY clock frequency is dependent on the current link speed.

Table 10. PHY Clock and Application Clock Frequencies

Link Speed PHY Clock Frequency Application Clock Frequency

Gen1 is supported only via link down-training and

not natively. Hence, the application clock frequency
depends on the configuration you choose in the IP
Gen1 125 MHz
Parameter Editor. For example, if you choose a Gen3
configuration, the application clock frequency is 250
MHz.

Gen2 is supported only via link down-training and

not natively. Hence, the application clock frequency
depends on the configuration you choose in the IP
Gen2 250 MHz
Parameter Editor. For example, if you choose a Gen3
configuration, the application clock frequency is 250
MHz.

Gen3 500 MHz 250 MHz

350 MHz / 400 MHz (Intel Stratix 10 DX)

Gen4 1000 MHz
350 MHz / 400 MHz / 500 MHz (Intel Agilex)

2.1.2. Refclk
P-Tile has two reference clock inputs at the package level, refclk0 and refclk1.
You must connect a 100 MHz reference clock source to these two inputs. Depending
on the port mode, you can drive the two refclk inputs using either a single clock
source or two independent clock sources.

In 1x16 and 4x4 modes, drive the refclk inputs with a single clock source (through
a fanout buffer) as shown in the figure below.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

14
2. IP Architecture and Functional Description
UG-20225 | 2021.02.18

Figure 3. Using a Single 100 MHz Clock Source in 1x16 and 4x4 Modes

PCIe x4 PCIe x8/ PCIe x16/ PCIe x4

x4 x8/x4
(Port 3) (Port 1) (Port 0) (Port 2)

Lane 15 PHY3 PHY2 PHY1 PHY0

Lane 12
Lane 11

Lane 8

Lane 3

Lane 0
Lane 7

Lane 4
(x4) (x4) (x4) (x4)

Refclk distribution on the package substrate

Refclk1 Refclk0
Fanout Buffer

100MHz

In 2x8 mode, you can drive the refclk inputs with either a single 100 MHz clock
source as shown above, or two independent 100 MHz sources (see the figure below)
depending on your system architecture. For example, if your system has each x8 port
connected to a separate CPU/Root Complex, it may be required to drive these refclk
inputs using independent clock sources. In that case, it is strongly recommended that
the refclk0 input for Port 0 be always running because it feeds the reference clock
for the P-Tile core PLL that controls the data transfers between the P-Tile and FPGA
fabric via the EMIB. If this clock goes down, Port 0 link will go down and Port 1 will not
be able to communicate with the FPGA fabric. Following are the guidelines for
implementing two independent refclks in 2x8 mode:
• If the link can handle two separate reference clocks, drive the refclk0 of P-Tile
with the on-board free-running oscillator.
• If the link needs to use a common reference clock, then PERST# needs to indicate
the stability of this reference clock. If this reference clock goes down, the entire P-
Tile must be reset.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

15
 2. IP Architecture and Functional Description
UG-20225 | 2021.02.18

Figure 4. Using Independent 100 MHz Clock Sources in 2x8 Mode

PCIe x4 PCIe x8/ PCIe x16/ PCIe x4

x4 x8/x4
(Port 3) (Port 1) (Port 0) (Port 2)

PHY3 PHY2 PHY1 PHY0

Lane 15

Lane 12
Lane 11

Lane 8

Lane 3

Lane 0
Lane 7

Lane 4
(x4) (x4) (x4) (x4)

Refclk distribution on the package substrate

Refclk1 Refclk0
100MHz 100MHz

2.1.3. Reset
There is only one PERST# (pin_perst_n) pin on P-Tile. Therefore, toggling
pin_perst_n will affect the entire P-Tile. If the P-Tile x16 port is bifurcated into two
x8 Endpoints, toggling pin_perst_n will affect both x8 Endpoints. To reset each port
individually, use the in-band mechanism such as Hot Reset and the Function-Level
Reset (FLR). Following are the guidelines for implementing the P-Tile pin_perst_n
reset signal:
• pin_perst_n is a "power good" indicator from the associated power domain (to
which P-Tile is connected). Also, it shall qualify that both the P-Tile refclk0 and
refclk1 are stable. If one of the reference clocks becomes stable later, deassert
pin_perst_n after this reference clock becomes stable.
• pin_perst_n assertion is required for proper Autonomous P-Tile functionality. In
Autonomous mode (enabled by default), P-Tile can successfully link up upon the
release of pin_perst_n regardless of the FPGA fabric configuration and will send
out CRS (Configuration Retry Status) until the FPGA fabric is configured and ready.

The following is an example where a single PERST# (pin_perst_n) is driven with

independent refclk0 and refclk1. In this example, the add-in card (FPGA and Soc)
is powered up first. P-Tile refclk0 is fed by the on-board free-running oscillator. P-
Tile refclk1 driven by the Host becomes stable later. Hence, the PERST# is
connected to the Host.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

16
2. IP Architecture and Functional Description
UG-20225 | 2021.02.18

Figure 5. Single PERST# Connection in Bifurcated 2x8 Mode

2.2. Functional Description

2.2.1. PMA/PCS
The P-Tile Avalon-ST IP for PCI Express contains Physical Medium Attachment (PMA)
and PCI Express Physical Coding Sublayer (PCIe PCS) blocks for handling the Physical
layer (PHY) packets. The PMA receives and transmits high-speed serial data on the
serial lanes. The PCS acts as an interface between the PMA and the PCIe controller,
and performs functions like data encoding and decoding, scrambling and
descrambling, block synchronization etc. The PCIe PCS in the P-Tile Avalon-ST IP for
PCI Express is based on the PHY Interface for PCI Express (PIPE) Base Specification
4.4.1.

In this IP, the PMA consists of up to four quads. Each quad contains a pair of transmit
PLLs and four SerDes lanes capable of running up to 16 GT/s to perform the various
TX and RX functions.

PLLA generates the required transmit clocks for Gen1/Gen2 speeds, while PLLB
generates the required clocks for Gen3/Gen4 speeds. For the x8 and x16 lane widths,
one of the quads acts as the master PLL source to drive the clock inputs for each of
the lanes in the other quads.

The PMA performs functions such as serialization/deserialization, clock data recovery,

and analog front-end functions such as Continuous Time Linear Equalizer (CTLE),
Decision Feedback Equalizer (DFE) and transmit equalization.

The transmitter consists of a 3-tap equalizer with one tap of pre-cursor, one tap of
main cursor and one tap of post-cursor.

The receiver consists of attenuation (ATT), CTLE, Voltage gain amplifier (VGA) and a
5-tap DFE blocks that are adaptive for Gen3/Gen4 speeds. RX Lane Margining is
supported by the PHY. The Lane Margining supports timing margining only. The
optional voltage margining is not supported. Timing margining capabilities/parameters
are as follows:
• Maximum Timing Offset: -0.2UI to +0.2UI.
• Number of timing steps: 9.
• Independent left and right timing margining is supported.
• Independent Error Sampler is not supported (lane margining may produce logical
errors in the data stream and cause the LTSSM to go to the Recovery state).

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

17
 2. IP Architecture and Functional Description
UG-20225 | 2021.02.18

The PHY layer uses a fixed 16-bit PCS-PMA interface width to output the PHY clock
(core_clk). The frequency of this clock is dependent on the current link speed. Refer
to Table 10 on page 14 for the frequencies at various link speeds.

Related Information
PHY Interface for PCI Express (PIPE) Base Specification

2.2.2. Data Link Layer Overview

The Data Link Layer (DLL) is located between the Transaction Layer and the Physical
Layer. It maintains packet integrity and communicates (by DLL packet transmission) at
the PCI Express link level.

The DLL implements the following functions:

• Link management through the reception and transmission of DLL Packets (DLLP),
which are used for the following functions:
— Power management of DLLP reception and transmission
— To transmit and receive ACK/NAK packets
— Data integrity through the generation and checking of CRCs for TLPs and
DLLPs
— TLP retransmission in case of NAK DLLP reception or replay timeout, using the
retry (replay) buffer
— Management of the retry buffer
— Link retraining requests in case of error through the Link Training and Status
State Machine (LTSSM) of the Physical Layer

Figure 6. Data Link Layer

To Transaction Layer To Physical Layer

Tx Transaction Layer Tx Arbitration

Packet Description & Data Transaction Layer
Packet Generator Tx Packets

Retry Buffer DLLP TX Datapath

Generator

Ack/Nack
Packets
Data Link Control Control
Power and Management & Status
Configuration Space
Management State Machine
Tx Flow Control Credit Information Function

Rx Flow Control Credit Information DLLP RX Datapath

Checker

Transaction Layer
Packet Checker Rx Packets
Rx Transation Layer
Packet Description & Data
Note:
(1) The L0s (Standby) or L1 (Low Power Standby) states are not supported.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

18
2. IP Architecture and Functional Description
UG-20225 | 2021.02.18

The DLL has the following sub-blocks:

• Data Link Control and Management State Machine—This state machine connects to
both the Physical Layer’s LTSSM state machine and the Transaction Layer. It
initializes the link and flow control credits and reports status to the Transaction
Layer.
• Power Management—This function handles the handshake to enter low power
mode. Such a transition is based on register values in the Configuration Space and
received Power Management (PM) DLLPs. For more details on the power states
supported by the P-Tile Avalon-ST IP for PCIe, refer to section Power Management
Interface on page 85.
• Data Link Layer Packet Generator and Checker—This block is associated with the
DLLP’s 16-bit CRC and maintains the integrity of transmitted packets.
• Transaction Layer Packet Generator—This block generates transmit packets,
including a sequence number and a 32-bit Link CRC (LCRC). The packets are also
sent to the retry buffer for internal storage. In retry mode, the TLP generator
receives the packets from the retry buffer and generates the CRC for the transmit
packet.
• Retry Buffer—The retry buffer stores TLPs and retransmits all unacknowledged
packets in the case of NAK DLLP reception. In case of ACK DLLP reception, the
retry buffer discards all acknowledged packets.
• ACK/NAK Packets—The ACK/NAK block handles ACK/NAK DLLPs and generates the
sequence number of transmitted packets.
• Transaction Layer Packet Checker—This block checks the integrity of the received
TLP and generates a request for transmission of an ACK/NAK DLLP.
• TX Arbitration—This block arbitrates transactions, prioritizing in the following
order:
— Initialize FC Data Link Layer packet
— ACK/NAK DLLP (high priority)
— Update FC DLLP (high priority)
— PM DLLP
— Retry buffer TLP
— TLP
— Update FC DLLP (low priority)
— ACK/NAK FC DLLP (low priority)

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

19
 2. IP Architecture and Functional Description
UG-20225 | 2021.02.18

2.2.3. Transaction Layer Overview

The following figure shows the major blocks in the P-Tile Avalon-ST IP for PCI Express
Transaction Layer:

Figure 7. P-Tile Avalon-ST IP for PCI Express Transaction Layer Block Diagram

Avalon-ST RX
Rx
User Avalon-MM
Data Link Layer
CPL Timeout +
CONFIG Logic RAS Physical Layer
CPL Timeout Avalon-MM

Avalon-ST TX
Tx

The RAS (Reliability, Availability, and Serviceability) block includes a set of features to
maintain the integrity of the link.

For example: Transaction Layer inserts an optional ECRC in the transmit logic and
checks it in the receive logic to provide End-to-End data protection.

When the application logic sets the TLP Digest (TD) bit in the Header of the TLP, the P-
Tile Avalon-ST IP for PCIe will append the ECRC automatically.

Note that in TLP Bypass mode, the PCIe Hard IP does not generate/check the ECRC
and will not remove it if the received TLP has the ECRC.

The TX block sends out the TLPs that it receives as-is. It also sends the information
about non-posted TLPs to the CPL Timeout Block for CPL timeout detection.

The P-Tile Avalon-ST IP for PCI Express RX block consists of two main blocks:
• Filtering block: This module checks if the TLP is good or bad and generates the
associated error message and completion. It also tracks received completions and
updates the completion timeout (CPL timeout) block.
• RX Buffer Queue: The P-Tile IP for PCIe has separate queues for posted/non-
posted transactions and completions. This avoids head-of-queue blocking on the
received TLPs and provides flexibility to extract TLPs according to the PCIe
ordering rules.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

20
2. IP Architecture and Functional Description
UG-20225 | 2021.02.18

Figure 8. P-Tile Avalon-ST IP for PCI Express RX Block Overview

Trash RX Buffer Queue Filter

TLP Filtering
P

Received CPL
Avalon-ST Routing NP Processing (*)
Data Logical
Message Link PHY
CPL Processing Layer Layer

MSG

ERR
MSG

User
Config CFG Data TX
Avalon-MM

Note: The Received CPL Processing block includes the CPL tracking mechanism.

Note: The Avalon-ST interface uses a split-bus architecture. In the x16 and x8
configurations, the 512-bit Avalon-ST data bus consists of two segments of 256-bit
data. This is done to improve the bandwidth efficiency of this interface. With this split-
bus architecture, two TLP packets can be transmitted or received in a single clock
cycle (e.g., if a TLP ends in the lower 256-bit segment,the next TLP can start in the
upper 256-bit segment in the same clock cycle). For more details, refer to Avalon-ST
Interface on page 51.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

21
 UG-20225 | 2021.02.18

Send Feedback

3. Parameters
This chapter provides a reference for all the parameters that are configurable in the
Intel Quartus Prime IP Parameter Editor for the P-Tile Avalon-ST IP for PCIe.

3.1. Top-Level Settings

Table 11. Top-Level Settings
Parameter Value Default Value Description

Gen4x16, Interface - 512- Select the following elements:

bit Lane data rate:
Gen3x16, Interface - 512- • Gen3, Gen4 are supported.
bit Gen4x16,
Lane width:
Hard IP Mode Gen4x8, Interface - 256-bit Interface - 512-
bit • x16 mode is for both Root Port
Gen3x8, Interface - 256-bit and Endpoint.
Gen4x4, Interface - 128-bit • x8 mode is for Endpoint only.
Gen3x4, Interface - 128-bit • x4 mode is for Root Port only.

Root Port
Native Endpoint
These are the available
options when Enable
TLP Bypass is set to
Port Mode False. If TLP Bypass Native Endpoint Specifies the port type.
Note: mode is enabled, refer
to the table Port Mode
Options in TLP Bypass
below for available port
mode options.

Enable the PHY Reconfiguration

Enable PHY Reconfiguration True/False False
Interface.

Select the frequency of the

Application clock. The options
available vary depending on the
setting of the Hard IP Mode
parameter.
500 MHz 400 MHz (for For Gen4 modes, the available
400 MHz Gen4 modes) clock frequencies are 500 MHz /
PLD Clock Frequency
350 MHz 250 MHz (for 400 MHz / 350 MHz (for Intel
250 MHz Gen3 modes) Agilex) and 400 MHz / 350 MHz
(for Intel Stratix 10 DX).
For Gen3 modes, the available
clock frequency is 250 MHz (for
Intel Agilex and Intel Stratix 10
DX).

Enable the TLP Bypass feature.

Enable TLP Bypass True/False False
Note:
continued...

Intel Corporation. All rights reserved. Agilex, Altera, Arria, Cyclone, Enpirion, Intel, the Intel logo, MAX, Nios,
Quartus and Stratix words and logos are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or
other countries. Intel warrants performance of its FPGA and semiconductor products to current specifications in ISO
accordance with Intel's standard warranty, but reserves the right to make changes to any products and services 9001:2015
at any time without notice. Intel assumes no responsibility or liability arising out of the application or use of any Registered
information, product, or service described herein except as expressly agreed to in writing by Intel. Intel
customers are advised to obtain the latest version of device specifications before relying on any published
information and before placing orders for products or services.
*Other names and brands may be claimed as the property of others.
3. Parameters
UG-20225 | 2021.02.18

Parameter Value Default Value Description

For configurations where multiple

ports are available, it is possible
to enable TLP Bypass on a per-
port basis. Refer to Table 12 on
page 23 for the available port
modes and configurations.

Enable the Separate Reference

Enable SRIS Mode True/False False Clock with Independent Spread
Spectrum Clocking (SRIS) feature.

Enabling this parameter reduces

the simulation time of Hot Reset
P-Tile Sim Mode True/False False tests by 5 ms.
Do not enable this option if
Note:
you need to run synthesis.

Enable the reset of PCS and

Controller in User Mode for
Endpoint and Bypass Upstream
modes.
When this parameter is True,
depending on the topology, new
signals (p<n>_pld_clrpcs_n)
are exported to the Avalon
Streaming interface.
When this parameter is False
Enable RST of PCS & (default), the IP internally ties off
True/False False
Controller these signals instead of exporting
them.
This feature is only
supported in the X8X8
Note:
Endpoint/Bypass Upstream
topology.
If you have more
questions regarding the
Note: bifurcation feature and its
usage, contact your
Application Engineer.

Table 12. Port Mode Options in TLP Bypass

Available Port Modes
Configuration
Port 0 Port 1 Port 2 Port 3

TLP Bypass On :
Downstream (Default)
1x16 (Gen4x16 or
N/A N/A N/A
Gen3x16)
TLP Bypass On :
Upstream

TLP Bypass On : TLP Bypass On :

Downstream (Default) Downstream (Default)

TLP Bypass On : TLP Bypass On :

2x8 (Gen4x8/ Upstream Upstream
Gen4x8 or Gen3x8/ N/A N/A
Gen3x8) TLP Bypass Off : TLP Bypass On :
Endpoint Upstream

TLP Bypass On : TLP Bypass On :

Upstream Downstream
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

23
 3. Parameters
UG-20225 | 2021.02.18

Available Port Modes

Configuration
Port 0 Port 1 Port 2 Port 3

TLP Bypass On : TLP Bypass Off :

Upstream Endpoint

TLP Bypass On : TLP Bypass On :

4x4 (Gen4x4/ TLP Bypass On : TLP Bypass On :
Downstream Downstream
Gen4x4 / Gen4x4/ Downstream (Default) Downstream (Default)
(Default) (Default)
Gen4x4 or Gen3x4/
Gen3x4 / Gen3x4/
TLP Bypass On : TLP Bypass On : TLP Bypass On : TLP Bypass On :
Gen3x4)
Upstream Upstream Upstream Upstream

Figure 9. Intel P-Tile Avalon-ST Top-Level IP Parameter Editor for a Gen4 x8 Hard IP in
Endpoint Mode

3.2. Core Parameters

Depending on which Hard IP Mode you choose in the Top-Level Settings tab, you
will see different tabs for setting the core parameters.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

24
3. Parameters
UG-20225 | 2021.02.18

Figure 10. Intel P-tile Avalon-ST Top-Level IP Parameter Editor for a x16 Hard IP Mode
If you choose a x16 mode (either Gen4 or Gen3), only the PCIe0 Settings tab will appear.

Note: You can enable the TLP Bypass mode in the Top-Level Settings tab of the IP
Parameter Editor as shown in the figure below:

Figure 11. Enabling TLP Bypass Mode

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

25
 3. Parameters
UG-20225 | 2021.02.18

Figure 12. Intel P-tile Avalon-ST Top-Level IP Parameter Editor for a x8 Hard IP Mode
If you choose a x8 mode (either Gen4 or Gen3), the PCIe0 Settings and PCIe1 Settings tabs will appear.

Figure 13. Intel P-tile Avalon-ST Top-Level IP Parameter Editor for a x4 Hard IP Mode
If you choose a x4 mode (either Gen4 or Gen3), the PCIe0 Settings, PCIe1 Settings, PCIe2 Settings and
PCIe3 Settings tabs will appear.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

26
3. Parameters
UG-20225 | 2021.02.18

3.2.1. System Parameters

Table 13. System Parameters
Parameter Value Default Value Description

Enable Multiple Physical Enable support for multiple

True/False False
Functions physical functions.

Note: If you set the Enable Multiple Physical Functions parameter to True, the
Multifunction and SR-IOV System Settings tab will appear to allow you to set the
number of physical functions and enable SR-IOV support.

3.2.2. Avalon Parameters

Table 14. Avalon Parameters
Parameter Value Default Value Description

When enabled, the Power

Management Interface and
Enable Power
Hard IP Status Interface are
Management Interface
True/False False exported. For more details,
and Hard IP Status
refer to section Power
Interface
Management Interface on
page 85.

Enable the support for

legacy interrupts. For more
Enable Legacy Interrupt True/False False details, refer to section
Legacy Interrupts on page
72.

Enable the support for parity

error checking. Parity errors
Enable Parity Error True/False True are indicated by outputs
rx_par_err_o and
tx_par_err_o.

Enable the Completion

Timeout Interface. For more
Enable Completion
True/False False details, refer to section
Timeout Interface
Completion Timeout
Interface on page 81.

Enable the Configuration

Intercept Interface. For
more details, refer to section
Configuration Intercept
Enable Configuration Interface (EP Only) on page
True/False False
Intercept Interface 91.
This parameter is
Note: only available in EP
mode.

Enable the Page Request

Service (PRS) Event
Interface. For more details,
Enable PRS Event True/False False refer to section Page
Request Service (PRS)
Interface (EP Only) on page
102.
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

27
 3. Parameters
UG-20225 | 2021.02.18

Parameter Value Default Value Description

This parameter is
Note: only available in EP
mode.

Enable the Error Interface.

For more details, refer to
Enable Error Interface True/False False
section Error Interface on
page 79.

When this parameter is

enabled, the byte parity
ports appear on the block
symbol. These byte parity
ports include:
Enable Byte Parity Ports rx_st_data_par_o,
True/False False rx_st_hdr_par_o,
on Avalon-ST Interface
rx_st_tlp_prfx_par_o,
tx_st_data_par_o,
tx_st_hdr_par_o, and
tx_st_tlp_prfx_par_o
ports.

3.2.3. Base Address Registers

Table 15. BAR Registers
Parameter Value Description

If you select 64-bit prefetchable memory, 2

contiguous BARs are combined to form a 64-bit
prefetchable BAR; you must set the higher
numbered BAR to Disabled.
Disabled Defining memory as prefetchable allows contiguous
64-bit prefetchable memory data to be fetched ahead. Prefetching memory is
advantageous when the requestor may require
BAR0 Type 64-bit non-prefetchable memory more data from the same region than was
32-bit non-prefetchable memory originally requested. If you specify that a memory
32-bit prefetchable memory is prefetchable, it must have the following 2
attributes:
• Reads do not have side effects such as
changing the value of the data read.
• Write merging is allowed.

Disabled
For a definition of prefetchable memory, refer to
BAR1 Type 32-bit non-prefetchable memory
the BAR0 Type description.
32-bit prefetchable memory

Disabled
64-bit prefetchable memory For a definition of prefetchable memory and a
description of what happens when you select the
BAR2 Type 64-bit non-prefetchable memory
64-bit prefetchable memory option, refer to the
32-bit non-prefetchable memory BAR0 Type description.
32-bit prefetchable memory

Disabled
For a definition of prefetchable memory, refer to
BAR3 Type 32-bit non-prefetchable memory
the BAR0 Type description.
32-bit prefetchable memory

Disabled
64-bit prefetchable memory For a definition of prefetchable memory and a
description of what happens when you select the
BAR4 Type 64-bit non-prefetchable memory
64-bit prefetchable memory option, refer to the
32-bit non-prefetchable memory BAR0 Type description.
32-bit prefetchable memory
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

28
3. Parameters
UG-20225 | 2021.02.18

Parameter Value Description

Disabled
For a definition of prefetchable memory, refer to
BAR5 Type 32-bit non-prefetchable memory
the BAR0 Type description.
32-bit prefetchable memory

Specifies the size of the address space accessible

BARn Size 128 Bytes - 16 EBytes to BARn when BARn is enabled.
n = 0, 1, 2, 3, 4 or 5

Disabled
4 KBytes - 12 bits
8 KBytes - 13 bits
16 KBytes - 14 bits
32 KBytes - 15 bits
64 KBytes - 16 bits
128 KBytes - 17 bits Specifies the size of the expansion ROM from 4
Expansion ROM
256 KBytes - 18 bits KBytes to 16 MBytes when enabled.
512 KBytes - 19 bits
1 MByte - 20 bits
2 MBytes - 21 bits
4 MBytes - 22 bits
8 MBytes - 23 bits
16 MBytes - 24 bits

3.2.4. Multi-function and SR-IOV

Table 16. Multi-function and SR-IOV
Parameter Value Default Value Description

Set the number of physical

functions. The IP core can
support 1 - 8 PFs. This
Total Physical Functions parameter is visible only if
1-8 1
(PFs) Enable multiple physical
functions is set to True
(under the PCIe Device
tab).

Enable SR-IOV Support True/False False Enable SR-IOV support.

Total Virtual Functions of Set the number of VFs to be

Physical Function 0 (PF0 0 - 2048 0 assigned to Physical
VFs) Function 0.

Enable VirtIO support. This

parameter is visible only if
Enable SR-IOV Support is
Enable VirtIO Support True/False False True and Enable multiple
physical functions is also
set to True (under the PCIe
Device tab).

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

29
 3. Parameters
UG-20225 | 2021.02.18

3.2.4.1. VirtIO Parameters

To enable VirtIO support, first enable the support for multiple physical functions in the
IP Parameter Editor as shown in the following screenshot:

Figure 14. Enable Multifunction Support

Make sure that SR-IOV support is also enabled:

Figure 15. Enable SR-IOV Support

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

30
3. Parameters
UG-20225 | 2021.02.18

Enable VirtIO support as shown in the screenshot below:

Figure 16. Enable VirtIO Support

Finally, you can configure the appropriate VirtIO capability parameters in the tabs
shown in the screenshot below:

Figure 17. Configure VirtIO Capability Parameters

The following table provides a reference for all the configurable high-level parameters
of the VirtIO block for P-Tile. Parameters below are dedicated to each core.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

31
 3. Parameters
UG-20225 | 2021.02.18

Table 17. VirtIO High-Level Parameters

Parameter Description Allowed Range Default Value

Total Physical Functions The number of supported 1-8 1

(PFs) Count Number Physical Functions.

Total Physical Functions Width of the supported 3 3

(PFs) Count Number Width Physical Functions number
count.

Total Virtual Functions Count Total number of VFs 0-2K 0

Number of PFs associated with PFs. Only
present when SR-IOV is
enabled.

Total Virtual Functions Count Width of the count of the 11 11

Number Width of PFs total number of VFs
associated with PFs. Only
present when SR-IOV is
enabled.

Virtual Functions Count Number of VFs associated 0-2K 0

Number associated with PF with PFs 0-7. The sum of all
0-7 the VF counts for PFs 0-7
cannot exceed the total
number of VFs.

Enable PF VirtIO Enable Physical Function 0-7 1’b1 / 1’b0 1’b0

VirtIO capability.

Enable VF VirtIO Enable VirtIO capability of 1’b1 / 1’b0 1’b0

VFs associated with PFs 0-7.

Base Address of VirtIO Start byte address of VirtIO 0x48 0x48

Common Configuration common configuration
Capability Structure capability structure for both
PFs and VFs.

The next table summarizes the parameters associated with the five VirtIO device
configuration structures:

Table 18. VirtIO Structure PCI Capabilities Parameters

Parameter Description Allowed Range Default Value

PF/VF VirtIO Common Configuration Structure Capability Parameters

PFs 0-7 Common Indicates BAR holding the 0-5 0

Configuration Structure BAR Common Configuration
Indicator Structure of PFs 0-7.

PFs 0-7 VFs Common Indicates BAR holding the 0-5 0

Configuration Structure BAR Common Configuration
Indicator Structure of VFs associated
with PFs 0-7.

PFs 0-7 Common Indicates starting position of 0-536870911 0

Configuration Structure Common Config Structure in
Offset within BAR a given BAR of PFs 0-7.

PFs 0-7 VFs Common Indicates starting position of 0-536870911 0

Configuration Structure BAR Common Config Structure in
Indicator a given BAR of VFs
associated with PFs 0-7.

PFs 0-7 Common Indicates length in bytes of 0-536870911 0

Configuration Structure Common Config Structure of
Length PFs 0-7.
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

32
3. Parameters
UG-20225 | 2021.02.18

Parameter Description Allowed Range Default Value

PFs 0-7 VFs Common Indicates length in bytes of 0-536870911 0

Configuration Structure Common Config Structure of
Length VFs associated with PFs 0-7.

PF/VF VirtIO Notifications Structure Capability Parameters

PFs 0-7 Notifications Indicates BAR holding the 0-5 0

Structure BAR Indicator Notifications Structure of PFs
0-7.

PFs 0-7 VFs Notifications Indicates BAR holding the 0-5 0

Structure BAR Indicator Notifications Structure of
VFs associated with PFs 0-7.

PFs 0-7 Notifications Indicates starting position of 0-536870911 0

Structure Offset within BAR Notifications Structure in
given BAR of PFs 0-7.

PFs 0-7 VFs Notifications Indicates starting position of 0-536870911 0

Structure BAR Indicator Notifications Structure in
given BAR of VFs associated
with PFs 0-7.

PFs 0-7 Notifications Indicates length in bytes of 0-536870911 0

Structure Length Notifications Structure of PFs
0-7.

PFs 0-7 VFs Notifications Indicates length in bytes of 0-536870911 0

Structure Length Notifications Structure of
VFs associated with PFs 0-7.

PFs 0-7 Notifications Indicates multiplier for 0-536870911 0

Structure Notify Off queue_notify_off in
Multiplier Notifications Structure of PFs
0-7.

PFs 0-7 VFs Notifications Indicates multiplier for 0-536870911 0

Structure Notify Off queue_notify_off in
Multiplier Notifications Structure of
VFs associated with PFs 0-7.

PF/VF VirtIO ISR Status Structure Capability Parameters

PFs 0-7 ISR Status Structure Indicates BAR holding the 0-5 0
BAR Indicator ISR Status Structure of PFs
0-7.

PFs 0-7 VFs ISR Status Indicates BAR holding the 0-5 0
Structure BAR Indicator ISR Status Structure of VFs
associated with PFs 0-7.

PFs 0-7 ISR Status Structure Indicates starting position of 0-536870911 0

Offset within BAR ISR Status Structure in
given BAR of PFs 0-7.

PFs 0-7 VFs ISR Status Indicates starting position of 0-536870911 0

Structure BAR Indicator ISR Status Structure in
given BAR of VFs associated
with PFs 0-7.

PFs 0-7 ISR Status Structure Indicates length in bytes of 0-536870911 0

Length ISR Status Structure of PFs
0-7.

PFs 0-7 VFs ISR Status Indicates length in bytes of 0-536870911 0

Structure Length ISR Status Structure of VFs
associated with PFs 0-7.

PF/VF VirtIO Device-Specific Configuration Structure Capability Parameters

continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

33
 3. Parameters
UG-20225 | 2021.02.18

Parameter Description Allowed Range Default Value

Enable PFs 0-7 VirtIO Device Enable PFs 0-7 VirtIO Ture / False False
Specific Capability Device-Specific
Configuration Structure
Capability.

Enable PFs 0-7 VFs VirtIO Enable VirtIO Device- Ture / False False
Device-Specific Capability Specific Configuration
Structure Capability of VFs
associated with PFs 0-7.

PFs 0-7 Device-Specific Indicates BAR holding the 0-5 0

Configuration Structure BAR Device-Specific
Indicator Configuration Structure of
PFs 0-7.

PFs 0-7 VFs Device-Specific Indicates BAR holding the 0-5 0

Configuration Structure BAR Device-Specific
Indicator Configuration Structure of
VFs associated with PFs 0-7.

PFs 0-7 Device-Specific Indicates starting position of 0-536870911 0

Configuration Structure Device-Specific
Offset within BAR Configuration Structure in
given BAR of PFs 0-7.

PFs 0-7 VFs Device-Specific Indicates starting position of 0-536870911 0

Configuration Structure BAR Device-Specific
Indicator Configuration Structure in
given BAR of VFs associated
with PFs 0-7.

PFs 0-7 Device-Specific Indicates length in bytes of 0-536870911 0

Configuration Structure Device-Specific
Length Configuration Structure of
PFs 0-7.

PFs 0-7 VFs Device-Specific Indicates length in bytes of 0-536870911 0

Configuration Structure Device-Specific
Length Configuration Structure of
VFs associated with PFs 0-7.

PF/VF VirtIO PCI Configuration Access Structure Capability Parameters

PFs 0-7 PCI Configuration Indicates BAR holding the 0-5 0

Access Structure BAR PCI Configuration Access
Indicator Structure of PFs 0-7.

PFs 0-7 VFs PCI Indicates BAR holding the 0-5 0

Configuration Access PCI Configuration Access
Structure BAR Indicator Structure of VFs associated
with PFs 0-7.

PFs 0-7 PCI Configuration Indicates Starting position of 0-536870911 0

Access Structure Offset PCI Configuration Access
within BAR Structure in given BAR of
PFs 0-7.

PFs 0-7 VFs PCI Indicates Starting position of 0-536870911 0

Configuration Access PCI Configuration Access
Structure BAR Indicator Structure in given BAR of
VFs associated with PFs 0-7.

PFs 0-7 PCI Configuration Indicates length in bytes of 0-536870911 0

Access Structure Length PCI Configuration Access
Structure of PFs 0-7.

PFs 0-7 VFs PCI Indicates length in bytes of 0-536870911 0

Configuration Access PCI Configuration Access
Structure Length Structure of VFs associated
with PFs 0-7.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

34
3. Parameters
UG-20225 | 2021.02.18

3.2.5. TLP Processing Hints (TPH)/Address Translation Services (ATS)

Capabilities
Table 19. TPH/ATS Capabilities
Parameter Value Default Value Description

Enable or disable Address

Translation Services (ATS)
capability.
Enable Address
Translation Services True/False False When ATS is enabled,
(ATS) senders can request and
cache translated addresses
using the RP memory space
for later use.

Enable or disable TLP

Processing Hints (TPH)
Enable TLP Processing capability.
True/False False
Hints (TPH) Using TPH may improve the
latency performance and
reduce traffic congestion.

3.2.6. PCI Express and PCI Capabilities Parameters

For each core (PCIe0/PCIe1/PCIe2/PCIe3), the PCI Express / PCI Capabilities tab
contains separate tabs for the device, PRS (Endpoint mode), MSI (Endpoint mode),
ACS capabilities (Root Port mode), slot (Root Port mode), MSI-X, and legacy interrupt
pin register parameters.

Figure 18. PCI Express / PCI Capabilities Parameters

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

35
 3. Parameters
UG-20225 | 2021.02.18

3.2.6.1. Device Capabilities

Table 20. Device Capabilities

Parameter Value Default Value Description

Specifies the maximum

payload size supported. This
128 bytes parameter sets the read-
Maximum payload sizes
256 bytes 512 bytes only value of the max
supported
512 bytes payload size supported field
of the Device Capabilities
register.

Enable Multiple Physical Enables multiple physical

True/False False
Functions functions.

When this option is True,

each function has its own
individual reset.
Required for all SR-IOV
Enable Function Level
True/False False functions.
Reset
This option appears only
when Enable Multiple
Physical Functions is set
to True.

3.2.6.2. Link Capabilities

Table 21. Link Capabilities

Parameter Value Default Value Description

Sets the read-only value of

the port number field in the
Link port number (Root Link Capabilities
0 - 255 1
Port only) register. This parameter is
for Root Ports only. It should
not be changed.

When this parameter is

True, it indicates that the
Endpoint uses the same
physical reference clock that
the system provides on the
connector. When it is False,
the IP core uses an
Slot clock configuration True/False True independent clock regardless
of the presence of a
reference clock on the
connector. This parameter
sets the Slot Clock
Configuration bit (bit 12) in
the PCI Express Link
Status register.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

36
3. Parameters
UG-20225 | 2021.02.18

3.2.6.3. Legacy Interrupt Pin Register

Table 22. Legacy Interrupts Parameters

Parameter Value Default Value Description

Enable Legacy Interrupts True/False False Enable Legacy Interrupts

for PF0 (INTx) for PF0 of PCIe0.

Set Interrupt Pin for PF0 NO INT NO INT When Legacy Interrupts are
INTA not enabled, the only option
available is NO INT.
INTA/INTB/INTC/INTD
When Legacy Interrupts are
enabled and multifunction is
disabled, the only option
available is INTA.
When Legacy Interrupts are
enabled and multifunction is
enabled, the options
available are INTA, INTB,
INTC and INTD.

3.2.6.4. MSI Capabilities

Table 23. MSI Capabilities

Parameter Value Default Value Description

Enables MSI functionality for

PF0.
If this parameter is True,
PF0 Enable MSI True/False False the Number of MSI
messages requested
parameter will appear
allowing you to set the
number of MSI messages.

Enables or disables MSI

PF0 MSI Extended Data
True/False False extended data capability for
Capable
PF0.

1
Sets the number of
2 messages that the
PF0 Number of MSI 4 application can request in
1 the multiple message
messages requested 8
16 capable field of the Message
Control register.
32

3.2.6.5. MSI-X Capabilities

Table 24. MSI-X Capabilities

Parameter Value Default Value Description

Enable MSI-X (Endpoint Enables the MSI-X

True/False False
only) functionality.

System software reads this

0x0 - 0x7FF (only values of
field to determine the MSI-X
MSI-X Table Size powers of two minus 1 are 0
table size <n>, which is
valid)
encoded as <n-1>.
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

37
 3. Parameters
UG-20225 | 2021.02.18

Parameter Value Default Value Description

For example, a returned

value of 2047 indicates a
table size of 2048. This field
is read-only.
Address offset:
0x068[26:16]

Points to the base of the

MSI-X table. The lower 3 bits
of the table BAR indicator
(BIR) are set to zero by
MSI-X Table Offset 0x0 - 0xFFFFFFFF 0
software to form a 64-bit
qword-aligned offset. This
field is read-only after being
programmed.

Specifies which one of a

function's BARs, located
beginning at 0x10 in
Configuration Space, is used
Table BAR indicator 0x0 - 0x5 0
to map the MSI-X table into
memory space. This field is
read-only after being
programmed.

Used as an offset from the

address contained in one of
the function's Base Address
registers to point to the base
Pending bit array (PBA) of the MSI-X PBA. The lower
0x0 - 0xFFFFFFFF 0
offset 3 bits of the PBA BIR are set
to zero by software to form
a 32-bit qword-aligned
offset. This field is read-only
after being programmed.

Specifies the function's Base

Address register, located
beginning at 0x10 in
Configuration Space, that
PBA BAR indicator 0x0 - 0x5 0
maps the MSI-X PBA into
memory space. This field is
read-only after being
programmed.

VF Table size 0x0 - 0x7FF (only values of 0 Sets the number of entries
powers of two minus 1 are in the MSI-X table for VFs.
valid) MSI-X cannot be disabled for
VFs. Set to 1 to save
resources.

3.2.6.6. Slot Capabilities

Note: This tab is visible in the Parameter Editor only if the Port Mode parameter in the
Top-Level Settings tab is set to Root Port.

Table 25. Slot Capabilities

Parameter Value Default Value Description

This parameter is only

supported in Root Port
Use Slot register True/False False mode. The slot capability is
required for Root Ports if a
slot is implemented on the
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

38
3. Parameters
UG-20225 | 2021.02.18

Parameter Value Default Value Description

port. Slot status is recorded

in the PCI Express
Capabilities register.

Specifies the scale used for

the slot power limit. The
following coefficients are
defined:
• 0 = 1.0x
• 1 = 0.1x
• 2 = 0.01x
Slot power scale 0-3 0 • 3 = 0.001x
The default value prior to
hardware and firmware
initialization is b’00. Writes
to this register also cause
the port to send the
Set_Slot_Power_Limit
message.

In combination with the Slot

power scale value,
Slot power limit 0 - 255 0 specifies the upper limit in
watts for the power supplied
by the slot.

Slot number 0 - 8191 0 Specifies the slot number.

3.2.6.7. Latency Tolerance Reporting (LTR)

This capability allows the P-Tile Avalon streaming IP, when operating in Endpoint
mode, to report the delay that it can tolerate when requesting service from the Host.
This information can help software optimize performance when the Endpoint needs a
fast response, or optimize system power when a fast response is not necessary.

Table 26. Latency Tolerance Reporting (LTR) Parameters

Parameter Value Default Value Description

PCIe0 Enable LTR True/False False Enable or disable LTR

capability for PCIe0.

3.2.6.8. Process Address Space ID (PASID)

Table 27. Process Address Space ID (PASID) Parameters

Parameter Value Default Value Description

PCIe0 PF0 Enable PASID True/False False Enable or disable PASID

capability for PCIe0 PF0.

PCIe0 PF0 Enable True/False False Enable or disable PASID

Execute Permission Execute Permission Support
Support for PCIe0 PF0.

PCIe0 PF0 Enable True/False False Enable or disable PASID

Privileged Mode Support Privileged Mode Support for
PCIe0 PF0.

PCIe0 PF0 Max PASID 0 - 20 0 Set the Max PASID Width for
Width PCIe0 PF0.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

39
 3. Parameters
UG-20225 | 2021.02.18

3.2.6.9. Device Serial Number Capability

Table 28. Device Serial Number Capability

Parameter Value Default Value Description

Enables the device serial

number capability. This is an
Enable Device Serial optional extended capability
True/False False
Number Capability that provides a unique
identifier for the PCIe
device.

3.2.6.10. Page Request Service (PRS)

Table 29. Page Request Service (PRS) Capability

Parameter Value Default Value Description

Enable or disable Page

Enable PRS True/False False Request Service (PRS)
capability.

3.2.6.11. Access Control Service (ACS) Capabilities

Table 30. ACS Capabilities for Physical Functions

Parameter Value Default Value Description

ACS defines a set of control

points within a PCI Express
Enable Access Control topology to determine
True/False False
Service (ACS) whether a TLP is to be
routed normally, blocked, or
redirected.

Enable ACS P2P Traffic Indicates if the component

True/False False
Support supports Peer to Peer Traffic.

Indicates if the component

implements ACS P2P Egress
Control.
Enable ACS P2P Egress
True/False False This parameter is visible
Control
only if Enable ACS P2P
Traffic Support is set to
True.

Indicates the number of bits

Enable ACS P2P Egress
0 - 255 0 in the ACS P2P Egress
Control Vector Size
Control Vector.

Table 31. ACS Capabilities for Virtual Functions

Parameter Value Default Value Description

ACS defines a set of control

points within a PCI Express
Enable Access Control topology to determine
True/False False
Service (ACS) whether a TLP is to be
routed normally, blocked, or
redirected.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

40
3. Parameters
UG-20225 | 2021.02.18

3.2.6.12. Power Management

Table 32. Power Management

Parameter Value Default Value Description

This design parameter

specifies the maximum
acceptable latency that the
application layer can tolerate
for any link between the
device and the root complex
to exit the L0s state. It sets
the read-only value of the
Endpoint L0s acceptable
latency field of the Device
Capabilities Register
(0x084).
Maximum of 64 ns This Endpoint does not
Maximum of 128 ns support the L0s or L1 states.
Maximum of 256 ns However, in a switched
Maximum of 512 ns system, there may be links
Enable L0s acceptable
Maximum of 64 ns connected to switches that
latency Maximum of 1 us have L0s and L1 enabled.
Maximum of 2 us This parameter is set to
Maximum of 4 us allow system configuration
No limit software to read the
acceptable latencies for all
devices in the system and
the exit latency for each link
to determine which links can
enable Active State Power
Management (ASPM).
This setting is disabled for
Root Ports.
The default value of this
parameter is 64 ns. This is
the safest setting for most
designs.

This value indicates the

acceptable latency that an
Endpoint can withstand in
the transition from the L1
state to L0 state. It is an
indirect measure of the
Endpoint’s internal buffering.
It sets the read-only value of
the Endpoint L1 acceptable
latency field of the Device
Maximum of 1 us
Capabilities Register.
Maximum of 2 us
This Endpoint does not
Maximum of 4 us
support the L0s or L1 states.
Endpoint L1 acceptable Maximum of 8 us However, a switched system
Maximum of 1 us
latency Maximum of 16 us may include links connected
Maximum of 32 us to switches that have L0s
and L1 enabled. This
Maximum of 64 us
parameter is set to allow
No limit system configuration
software to read the
acceptable latencies for all
devices in the system and
the exit latency for each link
to determine which links can
enable Active State Power
Management (ASPM).
This setting is disabled for
Root Ports.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

41
 3. Parameters
UG-20225 | 2021.02.18

3.2.6.13. Vendor Specific Extended Capability (VSEC) Registers

Table 33. VSEC Register

Parameter Value Default Value Description

Vendor Specific Extended Enables the Vendor Specific

0/1 0
Capability Extended Capability (VSEC).

Sets the read-only value of

the 16-bit User ID register
User ID register from the
from the Vendor Specific
Vendor Specific Extended 0 - 65534 0
Extended Capability. This
Capability
parameter is only valid for
Endpoints.

When this parameter is set

to 1, the IP core drops
vendor Type 0 messages
Drops Vendor Type0 while treating them as
0/1 0 Unsupported Requests (UR).
Messages
When it is set to 0, the IP
core passes these messages
on to the user logic.

When this parameter is set

to 1, the IP core silently
drops vendor Type 1
Drops Vendor Type1 messages.
0/1 0
Messages
When it is set to 0, the IP
core passes these messages
on to the user logic.

3.2.7. Device Identification Registers

The following table lists the default values of the Device ID registers. You can use the
parameter editor to change the values of these registers.

Table 34. Device ID Registers

Register Name Range Default Value Description

Sets the read-only value of the

Vendor ID register. This parameter
cannot be set to 0xFFFF per the
Vendor ID 16 bits 0x00001172 PCI Express Base Specification.
Set your own Vendor ID by
Note:
changing this parameter.
Address offset: 0x000.

Sets the read-only value of the

Device ID register. This register is
Device ID 16 bits 0x00000000 only valid in the Type 0 (Endpoint)
Configuration Space.
Address offset: 0x000.

Sets the read-only value of the

Revision ID 8 bits 0x00000001 Revision ID register.
Address offset: 0x008.

Sets the read-only value of the

Class Code 24 bits 0x00FF0000 Class Code register.
Address offset: 0x008.
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

42
3. Parameters
UG-20225 | 2021.02.18

Register Name Range Default Value Description

This parameter cannot be set to

0x0 per the PCI Express Base
Specification.

Sets the read-only value of the

Subsystem Vendor ID register in
the PCI Type 0 Configuration
Space. This parameter cannot be
Subsystem Vendor ID 16 bits 0x00000000 set to 0xFFFF per the PCI Express
Base Specification. This value is
assigned by PCI-SIG to the device
manufacturer.
Address offset: 0x02C.

Sets the read-only value of the

Subsystem Device ID register in
Subsystem Device ID 16 bits 0x00000000 the PCI Type 0 Configuration
Space.
Address offset: 0x02C.

3.2.8. Configuration, Debug and Extension Options

Table 35. Configuration, Debug and Extension Options
Parameter Value Default Value Description

Specifies the Gen 3

requested phase 2/3 far-end
Gen 3 Requested
TX preset vector. Choosing a
equalization far-end TX 0 - 65535 0x00000004
value different from the
preset vector
default is not recommended
for most designs.

Specifies the Gen 4

requested phase 2/3 far-end
Gen 4 Requested
TX preset vector. Choosing a
equalization far-end TX 0 - 65535 0x00000270
value different from the
preset vector
default is not recommended
for most designs.

When selected, RX buffer

limit ports are exported
allowing you to control the
Enable RX Buffer Limit buffer limits for RX Posted,
True/False False
Ports Non-Posted and CplD
packets. Otherwise, the
Maximum Buffer Size is
used.

If this parameter is True

(default), the refclk1 is
stable after pin_perst and
is free-running. This
parameter must be set to
True for Type A/B/C
systems.
If this parameter is False,
Port 1 REFCLK Init Active True/False True
refclk1 is only available
later in User Mode. This
parameter must be set to
False for Type D systems.
This parameter is only
available in the PCIe1
Settings tab for a X8X8
topology.
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

43
 3. Parameters
UG-20225 | 2021.02.18

Parameter Value Default Value Description

If you have more

questions regarding
the bifurcation
Note:
feature and its
usage, contact your
Application Engineer.

Enable the P-Tile Debug

Toolkit for JTAG-based
Enable Debug Toolkit True/False False
System Console debug
access.

Enable HIP dynamic Enable the user Hard IP

reconfiguration of PCIe True/False False reconfiguration Avalon-MM
registers interface.

Figure 19. Configuration, Debug and Extension Parameters (with Debug Toolkit
enabled)

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

44
 UG-20225 | 2021.02.18

Send Feedback

4. Interfaces
This section focuses mainly on the signal interfaces that the P-Tile IP for PCIe uses to
communicate with the Application Layer in the FPGA fabric core. However, it also
briefly covers the Serial Data Interface, which allows the IP to communicate with the
link partner across the PCIe link.

4.1. Overview
You can determine which core each interface in this section belongs to by looking at
the prefixes in the signal names:
• p0 : x16 core
• p1 : x8 core
• p2 : x4_0 core
• p3 : x4_1 core

Figure 20 on page 47 shows the top-level signals of this IP. Note that the signal
names in the figure will get the appropriate prefix pn (where n = 0, 1, 2 or 3)
depending on which of the three supported configurations (1x16, 2x8, or 4x4) the P-
Tile Avalon-ST IP for PCI Express is in.

As an example, the rx_st_data_o bus can take on the following names:

• In the 1x16 configuration, only the x16 core is active. In this case, this bus
appears as p0_rx_st_data_o[511:0].
• In the 2x8 configuration, both the x16 core and x8 core are active. In this case,
this bus is split into p0_rx_st_data_o[255:0] and
p1_rx_st_data_o[255:0].
• In the 4x4 configuration, all four cores are active. In this case, this bus is split into
p0_rx_st_data_o[127:0], p1_rx_st_data_o[127:0],
p2_rx_st_data_o[127:0] and p3_rx_st_data_o[127:0].

The only cases where the interface signal names do not get the pn prefixes are the
interfaces that are common for all the cores, like the PHY reconfiguration interface,
clocks and resets. For example, there is only one xcvr_reconfig_clk that is shared
by all the cores.

You can enable the PHY reconfiguration interface from the Top Level Settings in the
GUI.

Intel Corporation. All rights reserved. Agilex, Altera, Arria, Cyclone, Enpirion, Intel, the Intel logo, MAX, Nios,
Quartus and Stratix words and logos are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or
other countries. Intel warrants performance of its FPGA and semiconductor products to current specifications in ISO
accordance with Intel's standard warranty, but reserves the right to make changes to any products and services 9001:2015
at any time without notice. Intel assumes no responsibility or liability arising out of the application or use of any Registered
information, product, or service described herein except as expressly agreed to in writing by Intel. Intel
customers are advised to obtain the latest version of device specifications before relying on any published
information and before placing orders for products or services.
*Other names and brands may be claimed as the property of others.
 4. Interfaces
UG-20225 | 2021.02.18

Each of the cores has its own Avalon-ST interface to the user logic. The number of IP-
to-User Logic interfaces exposed to the FPGA fabric are different based on the
configuration modes:

Table 36. IP to FPGA Fabric Interfaces Summary

Avalon-ST Data Width (each Header Width TLP Prefix Width Application Clock
Mode
Interface Count Interface) (each Interface) (each Interface) Frequency

350 MHz / 400

MHz (Intel Stratix
Gen4 x16 EP/RP 10 DX)
1 512-bit 256-bit 64-bit
mode 350 MHz / 400
MHz / 500 MHz
(Intel Agilex)

Gen3 x16 EP/RP

1 512-bit 256-bit 64-bit 250 MHz
mode

350 MHz / 400

MHz (Intel Stratix
Gen4 x8 x8 EP 10 DX)
2 256-bit 128-bit 32-bit
mode 350 MHz / 400
MHz / 500 MHz
(Intel Agilex)

Gen3 x8 x8 EP
2 256-bit 128-bit 32-bit 250 MHz
mode

350 MHz / 400

MHz (Intel Stratix
Gen4 x4 x4 x4 x4 10 DX)
4 128-bit 128-bit 32-bit
RP mode 350 MHz / 400
MHz / 500 MHz
(Intel Agilex)

Gen3 x4 x4 x4 x4
4 128-bit 128-bit 32-bit 250 MHz
RP mode

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

46
4. Interfaces
UG-20225 | 2021.02.18

Figure 20. P-Tile Avalon-ST IP for PCI Express Top-Level Signals

P-Tile Avalon-ST Hard IP for PCIe*
rx_st_data_o[128*<w>-1:0] hip_reconfig_clk
rx_st_hdr_o[128*<n>-1:0] hip_reconfig_address[20:0]
rx_st_tlp_prfx_o[32*<n>-1:0]
rx_st_sop_o[<n>-1:0] hip_reconfig_read
rx_st_eop_o[<n>-1:0] hip_reconfig_readdata[7:0] Hard IP
rx_st_valid_o[<n>-1:0] hip_reconfig_readdatavalid Reconfiguration
rx_par_err_o hip_reconfig_write
Avalon-ST RX rx_st_empty_o[<p>-1:0] hip_reconfig_writedata[7:0]
Interface rx_st_ready_i hip_reconfig_waitrequest
rx_st_bar_range_o[3*<n>-1:0]
rx_st_data_par_o[{128*<w>/8}-1:0] xcvr_reconfig_clk
rx_st_hdr_par_o[{128*<n>/8}-1:0] xcvr_reconfig_address[25:0]
rx_st_tlp_prfx_o[4*<n>-1:0] xcvr_reconfig_read
rx_buffer_limit_i[11:0] xcvr_reconfig_readdata[7:0] PHY
rx_buffer_limit_tdm_idx_i[1:0] xcvr_reconfig_readdatavalid Reconfiguration
xcvr_reconfig_write
xcvr_reconfig_writedata[7:0]
xcvr_reconfig_waitrequest
tx_st_data_i[128*<w>-1:0]
tx_st_hdr_i[128*<n>-1:0]
tx_st_tlp_prfx_i[32*<n>-1:0] pm_state_o[2:0]
tx_st_sop_i[<n>-1:0] pm_dstate_o[31:0]
tx_st_eop_i[<n>-1:0] apps_pm_xmt_pme_i[7:0]
tx_st_valid_i[<n>-1:0] apps_ready_entr_l23_i Power
Avalon-ST TX apps_pm_xmt_turnoff_i Managementt
tx_st_ready_o
Interface app_init_rst_i
tx_par_err_o
tx_st_data_par_i[{128*<w>/8}-1:0] app_req_retry_en_i
tx_st_hdr_par_i[{128*<n>/8}-1:0]
tx_st_tlp_prfx_par_i[4*<n>-1:0]
tx_cdts_limit_o[15:0]
tx_cdts_limit_tdm_idx_o[2:0] rx_n_in[<b>-1:0]
rx_p_in[<b>-1:0] Serial Data
refclk[1:0] tx_n_out[<b>-1:0] Interface
Clocks coreclkout_hip tx_p_out[<b>-1:0]
reset_status_n
Resets pin_perst_n
ninit_done flr_rcvd_pf_o[7:0]
flr_rcvd_vf_o
sys_pwr_fault_det_i flr_rcvd_pf_num_o[2:0]
sys_pre_det_chged_i flr_rcvd_vf_num_o[10:0]
sys_mrl_sensor_chged_i Function Level Reset
flr_completed_pf_i[7:0]
sys_aux_pwr_det_i
Hot Plug flr_completed_vf_i
sys_cmd_cpled_int_i
Interface sys_mrl_sensor_state_i flr_completed_pf_num_i[2:0]
sys_pre_det_state_i flr_completed_vf_num_i[10:0]
sys_atten_button_pressed_i
sys_eml_interlock_engaged_i link_up_o
dl_up_o Hard IP Status
ltssm_state_o[5:0] Interface (*)
tl_cfg_ctl_o[15:0]
Configuration tl_cfg_add_o[4:0]
Output Interface
tl_cfg_func_o[2:0]

app_err_valid_i
serr_out_o
Error Interface hip_enter_err_mode_o
app_err_hdr_i[31:0] cii_req_o
app_err_info_i[12:0] cii_hdr_poisoned_o
cii_hdr_first_be_o[3:0]
app_err_func_num_i[2:0]
cii_func_num_o[2:0]
cii_wr_vf_active_o Configuration
cpl_timeout_o
cii_vf_num_o[10:0] Intercept
cpl_timeout_avmm_read_i
cpl_timeout_avmm_readdata_o[7:0] cii_wr_o Interface
Completion cpl_timeout_avmm_readdata_valid_o cii_addr_o[9:0]
Timeout cpl_timeout_avmm_write_i cii_dout_o[31:0]
Interface cpl_timeout_avmm_writedata_i[7:0] cii_override_en_i
cpl_timeout_avmm_clk_i cii_override_din_i[31:0]
cpl_timeout_avmm_waitrequest_o cii_halt_i
cpl_timeout_avmm_addr_i[20:0] app_int_i[7:0]
int_status_o[7:0]
Page Request prs_event_valid_i msi_pnd_func_i[2:0] Interrupt
msi_pnd_addr_i[1:0] Interface
Service(PRS) prs_event_func_i[2:0]
Event Interface prs_event_i[1:0] msi_pnd_byte_i[7:0]

Note: (*) : The Hard IP Status Interface is only available if the Power Management Interface
is enabled in the IP Parameter Editor.

Note: The table below shows the variables that are used to define the bus indices for top-
level signal busses shown in the top-level block diagram above. The values of these
variables change depending on which configuration is active (1x16, 2x8 or 4x4). For
example, for the 4x4 configuration, using w=1 and n=1 will give Avalon-ST RX bus
widths of p0_rx_st_data_o[127:0], p0_rx_st_hdr_o[127:0] and
p0_rx_st_tlp_prfx_o[31:0].

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

47
 4. Interfaces
UG-20225 | 2021.02.18

Table 37. Variables Used in the Bus Indices

Variable 1x16 Configuration 2x8 Configuration 4x4 Configuration

w 4 2 1

n 2 1 1

p 6 3 2

b 16 8 4

4.2. Clocks and Resets

4.2.1. Interface Clock Signals

Table 38. Interface Clock Signals
Name I/O Description EP/RP/BP Clock Frequency

Native Gen3: 250 MHz

This clock drives the Application Native Gen4: 350
Layer. MHz / 400 MHz (Intel
coreclkout_hip O The frequency depends on the data EP/RP/BP Stratix 10 DX)
rate and the number of lanes being Native Gen4: 350
used. MHz / 400 MHz / 500
MHz (Intel Agilex)

These are the input reference

clocks for the IP core. These clocks
must be free-running.
refclk[1:0] I EP/RP/BP 100 MHz ± 300 ppm
For more details on how to connect
these clocks, refer to the section
Clock Sharing in Bifurcation Modes.

Clock for the hip_reconfig

interface. This is an Avalon-MM
interface. It is an optional interface 50 MHz - 125 MHz
that is enabled when the Enable (range)
p0_hip_reconfig_c
I HIP dynamic reconfiguration of EP/RP/BP
lk 100 MHz
PCIe read-only registers option
(recommended)
in the PCIe Configuration, Debug
and Extension Options tab is
enabled.

Clock for the PHY reconfiguration

interface. This is an Avalon-MM
interface. This optional interface is 50 MHz - 125 MHz
enabled when you turn on the (range)
xcvr_reconfig_clk I EP/RP/BP
Enable PHY reconfiguration 100 MHz
option in the Top-Level Settings (recommended)
tab. This interface is shared among
all the cores.

Avalon-MM clock for Completion

timeout interface. This interface is 50 MHz - 125 MHz
p0_cpl_timeout_av optional, and is enabled when the (range)
I EP/RP/BP
mm_clk Enable Completion Timeout 100 MHz
Interface option in the PCIe (recommended)
Avalon Settings tab is enabled.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

48
4. Interfaces
UG-20225 | 2021.02.18

4.2.2. Resets

4.2.2.1. Interface Reset Signals

Table 39. Interface Reset Signals

Signal Name Direction Clock EP/RP/BP Description

pin_perst_n Input Asynchronous EP/RP/BP This is an active-low

input to the PCIe Hard
IP, and implements
the PERST# function
defined by the PCIe
specification.

p<n>_pin_perst_n Output Asynchronous EP/RP/BP This is the PERST

where n = 0, 1, 2, output signal from the
3 Hard IP. It is derived
from the
pin_perst_n input
signal.

p<n>_reset_status Output Synchronous EP/RP/BP This active-low signal

_n where n = 0, 1, is held low until
2, 3 pin_perst_n has
been deasserted and
the PCIe Hard IP has
come out of reset.
This signal is
synchronous to
coreclkout_hip.
When port bifurcation
is used, there is one
such signal for each
Avalon-ST interface.
The signals are
differentiated by the
prefixes pn. This is a
per-port signal.

ninit_done Input Asynchronous EP/RP A "1" on this active-

low signal indicates
that the FPGA device
is not yet fully
configured. A "0"
indicates the device
has been configured
and is in normal
operating mode.

4.2.2.2. Function-Level Reset (FLR) Interface (EP Only)

FLR allows specific physical/virtual functions to be reset without affecting other

physical/virtual functions or the link they share. FLR can be enabled by checking the
check-box Enable Function Level Reset (FLR) in the PCIe Device tab of the PCIe
PCI Express / PCI Capabilities tab in the GUI.

This interface is only present in EP mode (for x16/x8 configurations).

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

49
 4. Interfaces
UG-20225 | 2021.02.18

Table 40. FLR Interface

Signal Name Direction Description Clock Domain EP/RP/BP

Active high signals. Once

asserted, the signals
remain high until the
Application Layer sets the
p0_flr_completed_pf
_i[7:0] high for the
associated function. The
Application Layer must
perform actions necessary
to clear any pending
p0_flr_rcvd_pf_o[7:0] O transactions associated coreclkout_hip EP
with the function being
reset. The Application
Layer must assert
p0_flr_completed_pf
_i[7:0] to indicate it
has completed the FLR
actions and is ready to re-
enable the PF. These
busses are differentiated
by the prefixes pn.

A one-cycle pulse
indicates that an FLR was
received from host
targeting a VF. When port
bifurcation is used, there
p0_flr_rcvd_vf_o O coreclkout_hip EP
is one such signal for
each Avalon-ST interface.
These signals are
differentiated by the
prefixes pn.

Parent PF number of the

VF undergoing FLR. When
port bifurcation is used,
there is one such bus for
p0_flr_rcvd_pf_num_o[2:0] O coreclkout_hip EP
each Avalon-ST interface.
These busses are
differentiated by the
prefixes pn.

VF number offset of the

VF undergoing FLR. When
port bifurcation is used,
there is one such bus for
p0_flr_rcvd_vf_num_o[10:0] O coreclkout_hip EP
each Avalon-ST interface.
These busses are
differentiated by the
prefixes pn.

One bit per PF. A one-

cycle pulse on any bit
indicates that the
application has completed
the FLR sequence for the
corresponding PF and is
p0_flr_completed_pf_i[7:0] I ready to be enabled. coreclkout_hip EP
When port bifurcation is
used, there is one such
bus for each Avalon-ST
interface. These busses
are differentiated by the
prefixes pn.
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

50
4. Interfaces
UG-20225 | 2021.02.18

Signal Name Direction Description Clock Domain EP/RP/BP

One-cycle pulse from the

application re-enables a
VF. When port bifurcation
is used, there is one such
p0_flr_completed_vf_i I coreclkout_hip EP
signal for each Avalon-ST
interface. These signals
are differentiated by the
prefixes pn.

Parent PF number of the

VF to re-enable. When
port bifurcation is used,
p0_flr_completed_pf_num_i[2:0 there is one such bus for
I coreclkout_hip EP
] each Avalon-ST interface.
These busses are
differentiated by the
prefixes pn.

VF number offset of the

VF to re-enable. When
port bifurcation is used,
p0_flr_completed_vf_num_i[10: there is one such bus for
I coreclkout_hip EP
0] each Avalon-ST interface.
These busses are
differentiated by the
prefixes pn.

4.3. Serial Data Interface

P-Tile natively supports 4, 8, or 16 PCIe lanes. Each lane includes a TX differential pair
and an RX differential pair. Data is striped across all available lanes.

Table 41. Serial Data Interface

Signal Name Direction Description

tx_p_out[<b>-1:0], O Transmit serial data outputs using the

tx_n_out[<b>-1:0] High Speed Differential I/O standard.

rx_p_in[<b>-1:0], I Receive serial data inputs using the

rx_n_in[<b>-1:0] High Speed Differential I/O standard.

4.4. Avalon-ST Interface

The P-tile PCIe Hard IP provides an Avalon-ST-like interface with separate header and
data to improve the bandwidth utilization.

The Avalon-ST interface has different data bus widths depending on the link width
configuration of the PCIe IP.

Table 42. Avalon-ST Interface Data and Header Bus Widths

PCIe Link Width Data Width (bits) Header Width (bits) TLP Prefix Width (bits)

x16 512 (2 x 256) 256 (2 x 128) 64 (2 x 32)

x8 256 128 32

x4 128 128 32

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

51
 4. Interfaces
UG-20225 | 2021.02.18

Note: • For the x16 configuration, two segments of 256-bit data and two segments of
128-bit header are available.
• x4 configuration is only present in Root Port mode.

4.4.1. TLP Header and Data Alignment for the Avalon-ST RX and TX
Interfaces
The TLP prefix, header and data are sent and received on the TX and RX interfaces.

The ordering of bytes in the header and data portions of packets is different. The first
byte of the header dword is located in the most significant byte of the dword. The first
byte of the data dword is located in the least significant byte of the dword on the data
bus.

Figure 21. Generic TLP Format

Figure 22. TLP Prefix, Header and Data on the RX and TX Interfaces of the P-Tile IP for
PCIe

4.4.2. Avalon-ST RX Interface

The Application Layer receives data from the Transaction Layer of the PCI Express IP
core over the Avalon-ST RX interface. The application must assert rx_st_ready_i
before transfers can begin.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

52
4. Interfaces
UG-20225 | 2021.02.18

This interface supports two rx_st_sop_o signals and two rx_st_eop_o signals per
cycle when the P-Tile IP is operating in a x16 configuration. It also does not follow a
fixed latency between rx_st_ready_i and rx_st_valid_o as specified by the
Avalon Interface Specifications.

The x16 core provides two segments with each one having 256 bits of data
(rx_st_data_o[511:256] and rx_st_data_o[255:0]), 128 bits of header
(rx_st_hdr_o[255:128] and rx_st_hdr_o[127:0]), and 32 bits of TLP prefix
(rx_st_tlp_prfx_o[63:32] and rx_st_tlp_prfx_o[31:0]). If this core is
configured in the 1x16 mode, both segments are used, so the data bus becomes a
512-bit bus rx_st_data_o[511:0]. The start of packet can appear in the upper
segment or lower segment, as indicated by the rx_st_sop_o[1:0] signals.

Note: To achieve the expected performance in Gen4 x16 mode, the user application needs to
take advantage of this segmented bus architecture. Otherwise, some performance
reduction may occur.

If this core is configured in the 2x8 mode, only the lower segment is used. In this
case, the data bus is a 256-bit bus rx_st_data_o[255:0].

Finally, if this core is configured in the 4x4 mode, only the lower segment is used and
only the MSB 128 bits of data are valid. In this case, the data bus is a 128-bit bus
rx_st_data_o[127:0].

The x8 core provides one segment with 256 bits of data, 128 bits of header and 32
bits of TLP prefix. If this core is configured in 4x4 mode, only the LSB 128 bits of data
are used.

The x4 core provides one segment with 128 bits of data, 128 bits of header and 32
bits of TLP prefix.

Table 43. Avalon-ST RX Interface

Signal Name Direction Description Clock Domain EP/RP/BP

This is the Receive data

bus. The Application
x16 PCIe configuration: Layer receives data from
rx_st_data_o[511:0] the Transaction Layer on
this bus.
x8 PCIe configuration:
O For TLPs with an end-of- coreclkout_hip EP/RP/BP
rx_st_data_o[255:0]
packet cycle in the lower
x4 configuration: 256 bits, the 512-bit
rx_st_data_o[127:0] interface supports a
start-of-packet cycle in
the upper 256 bits.

Specify the number of

dwords that are empty
x16: rx_st_empty_o[5:0] during cycles when the
rx_st_eop_o signals
x8: rx_st_empty_o[2:0] O coreclkout_hip EP/RP/BP
are asserted. These
x4: rx_st_empty_o[1:0] signals are not valid
when the rx_st_eop_o
signals are not asserted.

Indicates the Application

rx_st_ready_i I Layer is ready to accept coreclkout_hip EP/RP/BP
data.
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

53
 4. Interfaces
UG-20225 | 2021.02.18

Signal Name Direction Description Clock Domain EP/RP/BP

The readyLatency is 27
cycles.
If rx_st_ready_i is
deasserted by the
Application Layer on cycle
<n>, the Transaction
Layer in the PCIe Hard IP
continues to send traffic
up to <n>+
readyLatency cycles
after the deassertion of
rx_st_ready_i.
Once rx_st_ready_i
reasserts,
rx_st_valid_o
resumes data transfer
within readyLatency
cycles.
To achieve the best
performance, the
Application Layer must
include a receive buffer
large enough to avoid the
deassertion of
rx_st_ready_i.

Signals the first cycle of

the TLP when asserted in
conjunction with the
corresponding bit of
rx_st_valid_o[1:0].
rx_st_sop_o[1]: When
x16: rx_st_sop_o[1:0] asserted, signals the
O start of a TLP on coreclkout_hip EP/RP/BP
x8/x4: rx_st_sop_o
rx_st_data_o[511:25
6].
rx_st_sop_o[0]: When
asserted, signals the
start of a TLP on
rx_st_data_o[255:0].

Signals the last cycle of

the TLP when asserted in
conjunction with the
corresponding bit of
rx_st_valid_o[1:0].
rx_st_eop_o[1]: When
x16: rx_st_eop_o[1:0] asserted, signals the end
O of a TLP on coreclkout_hip EP/RP/BP
x8/x4: rx_st_eop_o
rx_st_data_o[511:25
6].
rx_st_eop_o[0]: When
asserted, signals the end
of a TLP on
rx_st_data_o[255:0].

These signals qualify the

x16: rx_st_valid_o[1:0] rx_st_data_o signals
O coreclkout_hip EP/RP/BP
x8/x4: rx_st_valid_o going into the Application
Layer.
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

54
4. Interfaces
UG-20225 | 2021.02.18

Signal Name Direction Description Clock Domain EP/RP/BP

This is the received

x16: rx_st_hdr_o[255:0] header, which follows the
O coreclkout_hip EP/RP/BP
x8/x4: rx_st_hdr_o[127:0] TLP header format of the
PCIe specifications.

This is the first TLP prefix

received, which follows
the TLP prefix format of
the PCIe specifications.
PASID is included.
These signals are valid
when the corresponding
rx_st_sop_o is
x16: rx_st_tlp_prfx_o[63:0] asserted.
O coreclkout_hip EP/RP/BP
x8/x4: rx_st_tlp_prfx_o[31:0] The TLP prefix uses a Big
Endian implementation
(i.e, the Fmt field is in
bits [31:29] and the Type
field is in bits [28:24]).
If no prefix is present for
a given TLP, that dword
(including the Fmt field)
is all zeros.

When asserted, these

signals indicate that the
received TLP is targeting
a virtual function. When
these signals are
deasserted, the received
TLP is targeting a
physical function and the
rx_st_func_num
x16: rx_st_vf_active_o[1:0] signals indicate the
O function number. EP
x8: rx_st_vf_active_o coreclkout_hip
These signals are valid
x4: NA
when the corresponding
rx_st_sop_o is
asserted.
These signals are
multiplexed with the
rx_st_hdr_o signals in
the x4 configuration.
These signals are valid in
Endpoint mode only.

Specify the target

physical function number
for the received TLP.
These signals are valid
when the corresponding
x16: rx_st_func_num_o[5:0] rx_st_sop_o is
x8: rx_st_func_num_o[2:0] O asserted. coreclkout_hip EP
x4: NA These signals are
multiplexed with the
rx_st_hdr_o signals in
the x4 configuration.
These signals are valid in
Endpoint mode only.

Specify the target VF

x16: rx_st_vf_num_o[19:0] number for the received
x8: rx_st_vf_num_o[10:0] O TLP. The application uses coreclkout_hip EP
x4: NA this information for both
request and completion
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

55
 4. Interfaces
UG-20225 | 2021.02.18

Signal Name Direction Description Clock Domain EP/RP/BP

TLPs. For a completion

TLP, these bits specify the
VF number of the
requester for this
completion TLP.
These signals are valid
when
rx_st_vf_active_o
and the corresponding
rx_st_sop_o are
asserted.
These signals are
multiplexed with the
rx_st_hdr_o signals in
the x4 configuration.
These signals are valid in
Endpoint mode only.

Specify the BAR for the

TLP being output.
For each BAR range, the
following encodings are
defined:
• 000: Memory BAR 0
• 001: Memory BAR 1
• 010: Memory BAR 2
x16: rx_st_bar_range_o[5:0] • 011: Memory BAR 3
O coreclkout_hip EP/RP
x8/x4: rx_st_bar_range_o[2:0] • 100: Memory BAR 4
• 101: Memory BAR 5
• 110: I/O BAR
• 111: Expansion ROM
BAR
These outputs are valid
when both rx_st_sop_o
and rx_st_valid_o are
asserted.

By default, the PCIe Hard

IP drops an errored TLP
(a malformed TLP, or a
TLP with an ECRC error
x16: rx_st_tlp_abort_o[1:0] or tag/requester ID (RID)
O mismatches). The PCIe coreclkout_hip EP/RP
x8/x4: rx_st_tlp_abort_o Hard IP asserts
rx_st_tlp_abort_o to
notify the application an
errored TLP has been
dropped.

Byte parity signals for

x16: rx_st_data_par_o[63:0] rx_st_data_o. These
x8: rx_st_data_par_o[31:0] O parity signals are not coreclkout_hip EP/RP/BP
x4: rx_st_data_par_o[15:0] available when ECC is
enabled.
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

56
4. Interfaces
UG-20225 | 2021.02.18

Signal Name Direction Description Clock Domain EP/RP/BP

Byte parity signals for

x16: rx_st_hdr_par_o[31:0] rx_st_hdr_o. These
O parity signals are not coreclkout_hip EP/RP/BP
x8/x4: rx_st_hdr_par_o[15:0] available when ECC is
enabled.

x16: Byte parity signals for

rx_st_tlp_prfx_par_o[7:0] rx_st_tlp_prfx_o.
O These parity signals are coreclkout_hip EP/RP/BP
x8/x4:
not available when ECC is
rx_st_tlp_prfx_par_o[3:0] enabled.

Asserted for a single

cycle to indicate that a
parity error was detected
in a TLP at the input of
the RX buffer. This error
is logged as an
uncorrectable internal
rx_par_err_o O coreclkout_hip EP/RP/BP
error in the VSEC
registers. If this error
occurs, you must reset
the Hard IP because
parity errors can leave
the Hard IP in an
unknown state.

Figure 23. Avalon-ST RX Packet Interface in 1x16 Mode

coreclkout_hip

p0_rx_st_ready_i

p0_rx_st_valid_o[0]

p0_rx_st_sop_o[0]

p0_rx_st_eop_0[0]

p0_rx_st_prfx_o[31:0] PRF0

p0_rx_st_hdr_o[127:0] HDR0

p0_rx_st_data_o[255:0] D0_0 D0_2 D0_4 D1_1 D1_3

p0_rx_st_empty_o[2:0] 0x1 0x7

p0_rx_st_valid_o[1]

p0_rx_st_sop_o[1]

p0_rx_st_eop_o[1]

p0_rx_st_prfx_o[63:32] PRF1

p0_rx_st_hdr_o[255:128] HDR1

p0_rx_st_data_o[511:256] D0_1 D0_3 D1_0 D1_2

p0_rx_st_empty_o[5:3]

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

57
 4. Interfaces
UG-20225 | 2021.02.18

Figure 24. Avalon-ST RX Packet Interface in 2x8 Mode

coreclkout_hip

pn_rx_st_ready_i

pn_rx_st_valid_o

pn_rx_st_sop_o

pn_rx_st_eop_o

pn_rx_st_prfx_o[31:0] PRF0 PRF1

pn_rx_st_hdr_o[127:0] HDR0 HDR1

pn_rx_st_data_o[255:0] D0_0 D0_1 D0_2 D0_3 D0_4 D1_0 D1_1 D1_2 D1_3

pn_rx_st_empty_o[2:0] 0x1 0x7

Note: In 2x8 mode, the pn prefix in the signal names is p0 and p1 for the two x8 ports.

Figure 25. Avalon-ST RX Packet Interface in 4x4 Mode

coreclkout_hip

pn_rx_st_ready_i

pn_rx_st_valid_0

pn_rx_st_sop_0

pn_rx_st_eop_0

pn_rx_st_prfx_0[31:0] PRF0 PRF1

pn_rx_st_hdr_0[127:0] HDR0 HDR1

pn_rx_st_data_0[127:0] D0_0 D0_1 D0_2 D0_3 D0_4 D0_5 D0_6 D0_7 D0_8 D0_9 D1_0 D1_1 D1_2 D1_3 D1_4 D1_5 D1_6

pn_rx_st_empty_0[1:0] 0x1 0x3

Note: In 4x4 mode, the pn prefix in the signal names is p0, p1, p2 and p3 for the four x4
ports.

Note: In the diagrams for the 1x16 or 2x8 modes, D0_0 represents a 256-bit block of data.
However, in the diagram for the 4x4 mode, D0_0 represents a 128-bit block of data.

4.4.3. Avalon-ST RX Interface rx_st_ready Behavior

The following timing diagram illustrates the timing of the RX interface when the
application throttles the P-Tile IP for PCIe by deasserting rx_st_ready_i. The
Transaction Layer in the P-Tile IP deasserts rx_st_valid_o within 27 cycles of the
rx_st_ready_i deassertion. It also reasserts rx_st_valid_o within 27 cycles after
rx_st_ready_i reasserts if there is more data to send. This behavior means that the
readyLatency of this interface is 27. Refer to the Avalon Interface Specifications for a
detailed definition of readyLatency. rx_st_data_o is held until the application is able
to accept it.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

58
4. Interfaces
UG-20225 | 2021.02.18

Figure 26. Avalon-ST RX Interface rx_st_ready Behavior

coreclkout_hip

rx_st_data_o[511:0]
rx_st_sop_o

rx_st_eop_o
rx_st_ready_i

rx_st_valid_o

27 clks 27 clks

Related Information
Avalon Interface Specifications

4.4.4. RX Flow Control Interface

The RX flow control interface provides information on the application's available RX
buffer space to the PCIe Hard IP in a time-division multiplexing (TDM) manner. It
reports the space available in number of TLPs.

The RX flow control interface is optional and disabled by default in the IP GUI. If
disabled, it indicates that there is no limit in the application RX buffer space.

Figure 27. RX Flow Control Interface's TDM Reporting of Credit Limits

Coreclkout_hip

rx_buffer_limit_i[11:0] P NP CPL P NP CPL P NP CPL

rx_buffer_limit_tdm_idx_i[1:0] 0 1 2 0 1 2 0 1 2

Flow control credits are available for the following TLP categories:
• Posted (P) transactions: TLPs that do not require a response.
• Non-posted (NP) transactions: TLPs that require a completion.
• Completions (CPL): TLPs that respond to non-posted transactions.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

59
 4. Interfaces
UG-20225 | 2021.02.18

Figure 28. Example of a Buffer Limits Update

Buffer
RX Flow P Extracted
NP MWR0
MWR1 MWR0 MWR1
Hard IP CPP

Buffer Limit Interface

Examples of Updated Buffer Limits after Extraction

PD = 0x0FF + 1 = 0x100
PH = 0x00E + 1 = 0x00F
PD= 0x100 + 1 = 0x101
PH = 0x00F + 1 = 0x010
Examples of Buffer Limit Initialization Values
CPLD= 0xFFF
CPLH= 0xFFF
PD = 0x0FF
PH = 0x00E
NPD = 0x00F
NPH = 0x00F

Table 44. Categorization of Transaction Types

TLP Type Category

Memory Write Posted

Memory Read
Non-posted
Memory Read Lock

I/O Read
Non-posted
I/O Write

Configuration Read
Non-posted
Configuration Write

Message Posted

Completion

Completion with Data

Completion
Completion Lock

Completion Lock with Data

Fetch and Add AtomicOp Non-posted

Table 45. RX Flow Control Interface

Signal Name Direction Description Clock Domain EP/RP/BP

When the RX Flow Control

Interface is enabled, the
application can use these
rx_buffer_limit_ signals for TLP flow control.
I coreclkout_hip EP/RP/BP
i[11:0] These signals indicate the
application RX buffer space
made available since reset/
initialization.
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

60
4. Interfaces
UG-20225 | 2021.02.18

Signal Name Direction Description Clock Domain EP/RP/BP

Initially, the signals are set

according to the buffer size
(in terms of the number of
TLPs the RX buffer can take).
The value of these signals
always increments and rolls
over. For example, if the
initial value is 0xfff, the
rx_buffer_limit_i[11:0
] value increments by 1 and
rolls over to 0x000 when one
received TLP exits the
application RX buffer.
If a TLP type is blocked due
to a lack of the corresponding
RX buffer space in the
application layer, other TLP
types may bypass it per the
PCIe transaction ordering
rules.
Note that the initial value of
rx_buffer_limit_i[11:0
] cannot be larger than 2048
TLPs.

These signals indicate the

type of buffer for the
corresponding
rx_buffer_limit_i[11:0
] signals. The Application
Layer should provide the
buffer limit information for all
the enabled ports in a TDM
rx_buffer_limit_t
I manner. The following coreclkout_hip EP/RP/BP
dm_idx_i[1:0] encodings are defined:
• 00: buffer limit for P type
of TLPs.
• 01: buffer limit for NP
type.
• 10: buffer limit for CPL
type.
• 11: reserved.

For more details on the usage of the scale factors, refer to Section 3.4.2 of the PCI
Express Base Specification, Rev. 4.0 Version 1.0.

4.4.5. Avalon-ST TX Interface

The Application Layer transfers data to the Transaction Layer of the PCI Express IP
core over the Avalon-ST TX interface. The Transaction Layer must assert
tx_st_ready_o before transmission begins. Transmission of a packet must be
uninterrupted when tx_st_ready_o is asserted.

This 512-bit interface supports two locations for the beginning of a TLP, bit[0] and
bit[256]. The interface supports multiple TLPs per cycle only when an end-of-packet
cycle occurs in the lower 256 bits.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

61
 4. Interfaces
UG-20225 | 2021.02.18

Note: This interface supports two tx_st_sop_i signals and two tx_st_eop_i signals per
cycle when the P-Tile IP is operating in a x16 configuration. It also does not follow a
fixed latency between the tx_st_ready_o and tx_st_valid_i[1:0] signals as
specified by the Avalon Interface Specifications. Data can be received any time within
the defined readyLatency, which is three coreclkout_hip cycles.

The x16 core provides two segments with each one having 256 bits of data
(tx_st_data_i[511:256] and tx_st_data_i[255:0]), 128 bits of header
(tx_st_hdr_i[255:128] and tx_st_hdr_i[127:0]), and 32 bits of TLP prefix
(tx_st_tlp_prfx_i[63:32] and tx_st_tlp_prfx_i[31:0]). If this core is
configured in the 1x16 mode, both segments are used, so the data bus becomes a
512-bit bus tx_st_data_i[511:0]. The start of packet can appear in the upper
segment or lower segment, as indicated by the tx_st_sop_i[1:0] signals.

Note: To achieve the expected performance in Gen4 x16 mode, the user application needs to
take advantage of this segmented bus architecture. Otherwise, some performance
reduction may occur.

If this core is configured in the 2x8 mode, only the lower segment is used. In this
case, the data bus is a 256-bit bus tx_st_data_i[255:0].

Finally, if this core is configured in the 4x4 mode, only the lower segment is used and
only the LSB 128 bits of data are valid. In this case, the data bus is a 128-bit bus
tx_st_data_i[127:0].

The x8 core provides one segment with 256 bits of data, 128 bits of header and 32
bits of TLP prefix. If this core is configured in 4x4 mode, only the LSB 128 bits of data
are used.

The x4 core provides one segment with 128 bits of data, 128 bits of header and 32
bits of TLP prefix.

Table 46. Avalon-ST TX Interface

Signal Name Direction Description Clock Domain EP/RP/BP

Application Layer data for

transmission. The Application Layer
must provide a properly formatted
TLP on the TX interface. Valid when
the corresponding tx_st_valid_i
signal is asserted.
x16: tx_st_data_i[511:0] The mapping of message TLPs is the
x8: tx_st_data_i[255:0] I same as the mapping of Transaction coreclkout_h
EP/RP/BP
Layer TLPs with 4-dword headers. ip
x16: tx_st_data_i[127:0] The number of data cycles must be
correct for the length and address
fields in the header. Issuing a
packet with an incorrect number of
data cycles results in the TX
interface hanging and becoming
unable to accept further requests.
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

62
4. Interfaces
UG-20225 | 2021.02.18

Signal Name Direction Description Clock Domain EP/RP/BP

There must be no Idle cycle

between the tx_st_sop_i
and tx_st_eop_i cycles
Note:
unless there is backpressure
with the deassertion of
tx_st_ready_o.

Indicate the first cycle of a TLP

when asserted in conjunction with
the corresponding bit of
tx_st_valid_i. For the x16
configuration:
• tx_st_sop_i[1]: When
asserted, indicates the start of a
TLP in
x16: tx_st_sop_i[1:0] coreclkout_h
I tx_st_data_i[511:256]. EP/RP/BP
x8/x4: tx_st_sop_i •
ip
tx_st_sop_i[0]: When
asserted, indicates the start of a
TLP in tx_st_data_i[255:0].
These signals are asserted for one
clock cycle per each TLP. They also
qualify the corresponding
tx_st_hdr_i and
tx_st_tlp_prfx_i signals.

Indicate the last cycle of a TLP when

asserted in conjunction with the
corresponding bit of
tx_st_valid_i. For the x16
configuration:
• tx_st_eop_i[1]: When
x16: tx_st_eop_i[1:0] asserted, indicates the end of a coreclkout_h
I TLP in EP/RP/BP
x8/x4: tx_st_eop_i ip
tx_st_data_i[511:256].
• tx_st_eop_i[0]: When
asserted, indicates the end of a
TLP in tx_st_data_i[255:0].
These signals are asserted for one
clock cycle per each TLP.

Qualify the corresponding data

segment of tx_st_data_i into the
IP core on ready cycles.
To facilitate timing closure, Intel
recommends that you register both
the tx_st_ready_o and
x16: tx_st_valid_i[1:0] coreclkout_h
I tx_st_valid_i signals. EP/RP/BP
x8/x4: tx_st_valid_i ip
There must be no Idle cycle
between the tx_st_sop_i
and tx_st_eop_i cycles
Note:
unless there is backpressure
with the deassertion of
tx_st_ready_o.

Indicates that the PCIe Hard IP is

ready to accept data for
transmission.
The readyLatency is three cycles. coreclkout_h
tx_st_ready_o O EP/RP/BP
If tx_st_ready_o is asserted by ip
the Transaction Layer in the PCIe
Hard IP on cycle <n>, then <n> +
readyLatency is a ready cycle,
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

63
 4. Interfaces
UG-20225 | 2021.02.18

Signal Name Direction Description Clock Domain EP/RP/BP

during which the Application may

assert tx_st_valid_i and
transfer data.
If tx_st_ready_o is deasserted by
the Transaction Layer on cycle <n>,
then the Application must deassert
tx_st_valid_i within the
readyLatency number of cycles
after cycle <n>.
tx_st_ready_o can be deasserted
inthe following conditions:
• The LTSSM is not ready.
• A Retry is in progress.
• There are not enough credits
available to send the request.
• The P-Tile Avalon-ST IP is busy
sending internally generated
TLPs.
• The internal P-Tile TX FIFO is
full.

When asserted, indicate an error in

the transmitted TLP. These signals
are asserted with tx_st_eop_i
and nullify a packet.
x16: tx_st_err_i[1:0] • tx_st_err_i[1]: When coreclkout_h
I EP/RP/BP
x8/x4: tx_st_err_i asserted, specifies an error in ip
tx_st_data_i[511:256].
• tx_st_err_i[0]: When
asserted, specifies an error in
tx_st_data_i[255:0].

This is the header to be transmitted,

which follows the TLP header format
of the PCIe specifications except for
the requester ID/completer ID fields
(tx_st_hdr_i[95:80]):
• tx_st_hdr_i[95:84]:
tx_st_vf_num[11:0]
x16: tx_st_hdr_i[255:0] • tx_st_hdr_i[83]: coreclkout_h
I EP/RP/BP
x8/x4: tx_st_hdr_i[127:0] tx_st_vf_active ip
• tx_st_hdr_i[82:80]:
tx_st_func_num[2:0]
These signals are valid when the
corresponding tx_st_sop_i signal
is asserted.
The header uses a Big Endian
implementation.

This is the TLP prefix to be

transmitted, which follows the TLP
prefix format of the PCIe
specifications. PASID is included.
x16:
tx_st_tlp_prfx_i[63:0] These signals are valid when the
coreclkout_h
I corresponding tx_st_sop_i signal EP/RP/BP
x8/x4: ip
is asserted.
tx_st_tlp_prfx_i[31:0]
The TLP prefix uses a Big Endian
implementation (i.e. the Fmt field is
in bits [31:29] and the Type field is
in bits [28:24]).
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

64
4. Interfaces
UG-20225 | 2021.02.18

Signal Name Direction Description Clock Domain EP/RP/BP

If no prefix is present for a given

TLP, that dword, including the Fmt
field, is all zeros.

Byte parity for tx_st_data_i. Bit

[0] corresponds to
tx_st_data_i[7:0], bit [1]
x16: corresponds to
tx_st_data_par_i[63:0] tx_st_data_i[15:8], and so on.
coreclkout_h
I By default, the PCIe Hard IP EP/RP/BP
x8: tx_st_data_par_i[31:0] ip
generates the parity for the TX data.
x4: tx_st_data_par_i[15:0] However, when ECC is off, the parity
can be passed in from the FPGA
core by setting the
k_pcie_parity_bypass register.

Byte parity for tx_st_hdr_i.

By default, the PCIe Hard IP
x16: tx_st_hdr_par_i[31:0] generates the parity for the TX coreclkout_h
x8/x4: I header. However, when ECC is off, EP/RP/BP
ip
tx_st_hdr_par_i[15:0] the parity can be passed in from the
FPGA core by setting the
k_pcie_parity_bypass register.

Byte parity for

tx_st_tlp_prfx_i.
x16: By default, the PCIe Hard IP
tx_st_tlp_prfx_par_i[7:0] generates the parity for the TX TLP coreclkout_h
I EP/RP/BP
x8/x4: prefix. However, when ECC is off, ip
tx_st_tlp_prfx_par_i[3:0] the parity can be passed in from the
FPGA core by setting the
k_pcie_parity_bypass register.

Asserted for a single cycle to

indicate a parity error during TX TLP
coreclkout_h
tx_par_err_o O transmission. The IP core transmits EP/RP/BP
TX TLP packets even when a parity
ip
error is detected.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

65
 4. Interfaces
UG-20225 | 2021.02.18

Figure 29. Avalon-ST TX Packet Interface in 1x16 Mode

coreclkout_hip

p0_tx_st_ready_o

p0_tx_st_valid_i[0]

p0_tx_st_sop_i[0]

p0_tx_st_eop_i[0]

p0_tx_st_tlp_prfx_i[31:0] PRF0

p0_tx_st_hdr_i[127:0] HDR0

p0_tx_st_data_i[255:0] D0_0 D0_2 D0_4 D1_1 D1_3

p0_tx_st_valid_i[1]

p0_tx_st_sop_i[1]

p0_tx_st_eop_i[1]

p0_tx_st_tlp_prfx_i[63:32] PRF1

p0_tx_st_hdr_i[255:128] HDR1

p0_tx_st_data_i[511:256] D0_1 D0_3 D1_0 D1_2

Figure 30. Avalon-ST TX Packet Interface in 2x8 Mode

coreclkout_hip

pn_tx_st_ready_0

pn_tx_st_valid_i

pn_tx_st_sop_i

pn_tx_st_eop_i

pn_tx_st_prfx_i[31:0] PRF0 PRF1

pn_tx_st_hdr_i[127:0] HDR0 HDR1

pn_tx_st_data_i[255:0] D0_0 D0_1 D0_2 D0_3 D0_4 D1_0 D1_1 D1_2 D1_3

Note: In 2x8 mode, the pn prefix in the signal names is p0 and p1 for the two x8 ports.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

66
4. Interfaces
UG-20225 | 2021.02.18

Figure 31. Avalon-ST TX Packet Interface in 4x4 Mode

coreclkout_hip

pn_tx_st_ready_o

pn_tx_st_valid_i

pn_tx_st_sop_i

pn_tx_st_eop_i

pn_tx_st_prfx_i[31:0] PRF0 PRF1

pn_tx_st_hdr_i[127:0] HDR0 HDR0

pn_tx_st_data_i[127:0] D0_0 D0_1 D0_2 D0_3 D0_4 D0_5 D0_6 D0_7 D0_8 D0_9 D1_0 D1_1 D1_2 D1_3 D1_4 D1_5 D1_6

Note: In 4x4 mode, the pn prefix in the signal names is p0, p1, p2 and p3 for the four x4
ports.

Note: In the diagrams for the 1x16 or 2x8 modes, D0_0 represents a 256-bit block of data.
However, in the diagram for the 4x4 mode, D0_0 represents a 128-bit block of data.

4.4.6. Avalon-ST TX Interface tx_st_ready Behavior

The following timing diagram illustrates the behavior of tx_st_ready_o, which is
deasserted to pause the data transmission to the Transaction Layer of the P-Tile IP for
PCIe, and then reasserted. The timing diagram shows a readyLatency of three cycles.
Refer to the Avalon Interface Specifications for a detailed definition of readyLatency.
The application deasserts tx_st_valid_i three cycles after tx_st_ready_o is
deasserted.

The application must not deassert tx_st_valid_i between tx_st_sop_i and

tx_st_eop_i on a ready cycle. For the definition of a ready cycle, refer to the Avalon
Interface Specifications.

Note: This is an additional requirement for the P-Tile IP for PCIe that is not compliant to the
Avalon-ST standard.

Figure 32. Avalon-ST TX Interface tx_st_ready Behavior

coreclkout_hip

tx_st_ready_o

tx_st_valid_i 3 clks
3 clks

tx_st_sop_i

tx_st_eop_i

Related Information
Avalon Interface Specifications

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

67
 4. Interfaces
UG-20225 | 2021.02.18

4.4.7. TX Flow Control Interface

Before a TLP can be transmitted, flow control logic verifies that the link partner's RX
port has sufficient buffer space to accept it. The TX Flow Control interface reports the
link partner's available RX buffer space to the Application. It reports the space
available in units called Flow Control credits for posted, non-posted and completion
TLPs (as defined in the RX Flow Control Interface section).

TX credit limit signals are provided in a TDM manner similar to how the RX credit limit
signals are provided.

Figure 33. TX Flow Control Interface's TDM Reporting of Credit Limits

coreclkout_hip

tx_cdts_limit_o[15:0] PH NPH CPLH PD NPD CPLD PH NPH CPLH

tx_cdts_limit_tdm_idx_o[2:0] 0 1 2 4 5 6 0 1 2

Figure 34. Example of Buffer Limits Update

Buffer

1DW MWR
1H
MWR0 TX MWR1 MWR0 TX MWR1

16DW
RP Hard IP
MWR
1H
Update Update
Credit RX Credit
Credit for MWRO
PD =0x100
PH = 0x00F
Initialization Values
Credit for MWR1
PD = 0x0FF
PD =0x104
PH = 0x00E
PH = 0x0D0
NPD = 0x00F
NPH = 0x00F

CPLD = 0xFFF
CPLH= 0xFFF

This example shows how this interface is updated when multiple MWr requests are
sent. The tx_cdts_limit_o[15:0] bus value is incremented when a TLP is
acknowledged by the receiver and will roll over when reaching 0xFFFF.

Table 47. TX Flow Control Interface

Signal Name Direction Description Clock Domain EP/RP/BP

Indicate the Flow

Control (FC) credit units
advertised by the
remote Receiver.
tx_cdts_limit_o[1
O These signals represent coreclkout_hip EP/RP/BP
5:0] the total number of FC
credits made available
by the Receiver since
Flow Control
initialization. Initially,
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

68
4. Interfaces
UG-20225 | 2021.02.18

Signal Name Direction Description Clock Domain EP/RP/BP

these signals indicate

the number of FC
credits available in the
remote Receiver. The
value of these signals
always increments and
rolls over.
For example, if the
remote Receiver
advertises an initial
Non-Posted Header
(NPH) FC credit of
0xFFFF, after it receives
a MRd request, the NPH
FC credits value
increments by 1 and
rolls over to 0x0000.
The
tx_cdts_limit_tdm_
idx_o[2:0] signals
determine the traffic
type.
When the traffic type is
header credit, only the
LSB 12 bits are valid.
Note that, in addition to
the TLPs transmitted by
the user application,
internally generated
TLPs also consume FC
credits.

Indicate the traffic type

for the
tx_cdts_limit_o[15
:0] signals.
This interface provides
credit limit information
for all enabled ports in a
TDM manner.
The following encodings
are defined:
• 000: P header credit
tx_cdts_limit_tdm limit
O • 001: NP header coreclkout_hip EP/RP/BP
_idx_o[2:0]
credit limit
• 010: CPL header
credit limit
• 011: reserved
• 100: P data credit
limit
• 101: NP data credit
limit
• 110: CPL data credit
limit
• 111: reserved

4.4.8. Tag Allocation

The P-Tile PCIe Hard IP supports the 10-bit tag Requester capability in the x16
Controller (Port 0) only. It supports up to 512 outstanding Non-Posted Requests
(NPRs) with valid tag values ranging from 256 to 767.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

69
 4. Interfaces
UG-20225 | 2021.02.18

The x8 (Port 1) and x4 Controllers (Port 2/3) don’t support the 10-bit tag Requester
capability, although they support the 10-bit Completer capability.

Both x8 and x4 Controllers can allow up to 256 outstanding NPRs with valid tag values
ranging from 0 to 255.

When enabling both 10-bit tags and 8-bit tags, the LSB 8 bits of the 8-bit tags cannot
be shared with the LSB 8 bits of the 10-bit tags. For example, if you want to use 64
tags as 8-bit tags and the rest of the tags as 10-bit tags, you can partition the tag
space as follows:
• 8-bit tags : 0 - 63
• 10-bit tags : 320 - 511, 576 - 767

Note that all PFs and their associated VFs share the same tag space. This means that
different PFs and VFs cannot have outstanding tags having the same tag values.

In the TLP bypass mode, there is no restriction on the tag allocation since the P-Tile
PCIe Hard IP does not do any tag management. Hence, 10-bit tags can be used
without any restriction across all the cores.

4.4.8.1. Completion Buffer Size

P-tile implements Completion (Cpl) buffers for header and data for each PCIe core. In
Endpoint mode, when Completion credits are infinite, user application needs to
manage the number of outstanding requests to prevent overflow and lost
Completions.

Table 48. Completion Buffer Size

Completion Buffer Depth Width

Port 0 Cpl header 1144 NA

Port 0 Cpl data 1444 256

Port 1 Cpl header 572 NA

Port 1 Cpl data 1444 128

Port 2 Cpl header 286 NA

Port 2 Cpl data 1444 64

Port 3 Cpl header 286 NA

Port 3 Cpl data 1444 64

4.5. Hard IP Status Interface

This interface includes the signals that are useful for debugging, such as the link
status signal, LTSSM state outputs, etc. These signals are available when the optional
Power Management interface is enabled.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

70
4. Interfaces
UG-20225 | 2021.02.18

Table 49. Hard IP Status Interface

Signal Name Direction Description Clock Domain EP/RP/BP

When asserted, this signal indicates the coreclkout_hi

link_up_o O EP/RP/BP
link is up. p

When asserted, this signal indicates the coreclkout_hi

dl_up_o O EP/RP/BP
Data Link (DL) Layer is active. p

ltssm_state_o[5:0 O Indicates the LTSSM state:

] • 6'h00: S_DETECT_QUIET
• 6'h01: S_DETECT_ACT
• 6'h02: S_POLL_ACTIVE
• 6'h03: S_POLL_COMPLIANCE
• 6'h04: S_POLL_CONFIG
• 6'h05: S_PRE_DETECT_QUIET
• 6'h06: S_DETECT_WAIT
• 6'h07: S_CFG_LINKWD_START
• 6'h08: S_CFG_LINKWD_ACCEPT
• 6'h09: S_CFG_LANENUM_WAIT
• 6'h0A: S_CFG_LANENUM_ACCEPT
• 6'h0B: S_CFG_COMPLETE
• 6'h0C: S_CFG_IDLE
• 6'h0D: S_RCVRY_LOCK
• 6'h0E: S_RCVRY_SPEED
• 6'h0F: S_RCVRY_RCVRCFG
• 6'h10: S_RCVRY_IDLE
coreclkout_hi
• 6'h11: S_L0 EP/RP/BP
p
• 6'h12: S_L0S
• 6'h13: S_L123_SEND_EIDLE
• 6'h14: S_L1_IDLE
• 6'h15: S_L2_IDLE
• 6'h16: S_L2_WAKE
• 6'h17: S_DISABLED_ENTRY
• 6'h18: S_DISABLED_IDLE
• 6'h19: S_DISABLED
• 6'h1A: S_LPBK_ENTRY
• 6'h1B: S_LPBK_ACTIVE
• 6'h1C: S_LPBK_EXIT
• 6'h1D: S_LPBK_EXIT_TIMEOUT
• 6'h1E: S_HOT_RESET_ENTRY
• 6'h1F: S_HOT_RESET
• 6'h20: S_RCVRY_EQ0
• 6'h21: S_RCVRY_EQ1
• 6'h22: S_RCVRY_EQ2
• 6'h23: S_RCVRY_EQ3

4.6. Interrupt Interface

The P-Tile Avalon-ST IP for PCI Express supports Message Signaled Interrupts (MSI),
MSI-X interrupts, and legacy interrupts. MSI and legacy interrupts are mutually
exclusive.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

71
 4. Interfaces
UG-20225 | 2021.02.18

The user application generates MSI which are single-Dword memory write TLPs to
implement interrupts. This interrupt mechanism conserves pins because it does not
use separate wires for interrupts. In addition, the single Dword provides flexibility for
the data presented in the interrupt message. The MSI Capability structure is stored in
the Configuration Space and is programmed using Configuration Space accesses.

The user application generates MSI-X messages which are single-Dword memory
writes. The MSI-X Capability structure points to an MSI-X table structure and an MSI-X
Pending Bit Array (PBA) structure which are stored in memory. This scheme is
different than the MSI Capability structure, which contains all the control and status
information for the interrupts.

Enable legacy interrupts by programming the Interrupt Disable bit (bit[10]) of

the Configuration Space Command to 1'b0. When legacy interrupts are enabled,
the IP core emulates INTx interrupts using virtual wires. The app_int_i ports control
legacy interrupt generation.

4.6.1. Legacy Interrupts

Legacy interrupts mimic the original PCI level-sensitive interrupts using virtual wire
messages. The P-tile IP for PCIe signals legacy interrupts on the PCIe link using
Message TLPs. The term INTx refers collectively to the four legacy interrupts,
INTA#,INTB#, INTC# and INTD#. The P-tile IP for PCIe asserts app_int_i to cause
an Assert_INTx Message TLP to be generated and sent upstream. A deassertion of
app_int_i, i.e a transition of this signal from high to low, causes a Deassert_INTx
Message TLP to be generated and sent upstream. To use legacy interrupts, you must
clear the Interrupt Disable bit, which is bit 10 of the Command Register in the
configuration header. Then, you must turn off the MSI Enable bit.

Table 50. Legacy Interrupt Interface

Signal Name Direction Description Clock Domain EP/RP/BP

When asserted, these signals

indicate an assertion of an INTx
message is requested. A
transition from high to low
x16/x8: indicates a deassertion of the
coreclkout_
app_int_i[7:0] I INTx message is requested. This EP
bus is for EP only. Each bit is
hip
x4: NA
associated with a corresponding
physical function. These signals
must be asserted for at least 8
cycles.

These signals drive legacy

interrupts to the Application
Layer in Root Port mode. The
source of the interrupt will be coreclkout_
int_status_o[7:0] O RP
logged in the Root Port Interrupt hip
Status registers in the Port
Configuration and Status
registers.

Figure 35. Generating an Assert_INTx Message TLP Using the app_int_i Signal
coreclkout_hip

app_int[0]

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

72
4. Interfaces
UG-20225 | 2021.02.18

app_int_i[0] is asserted for at least eight clock cycles to cause an Assert_INTx

Message TLP to be generated and sent upstream for physical function 0. For a multi-
functions implementation, app_int_i[0] is for physical function 0, app_int_i[1]
is for physical function 1 and so on. Deasserting an app_int_i signal by driving it
from high to low causes a Deassert_INTx Message TLP to be generated and sent
upstream.

4.6.2. MSI
MSI interrupts are signaled on the PCI Express link using a single dword Memory Write
TLP. The user application issues an MSI request (MWr) through the Avalon-ST interface
and updates the configuration space register using the MSI interface.

For more details on the MSI Capability Structure, refer to Figure 85 on page 188.

The Mask Bits register and Pending Bits register are 32 bits in length each, with each
potential interrupt message having its own mask bit and pending bit. If bit[0] of the
Mask Bits register is set, interrupt message 0 is masked. When an interrupt message
is masked, the MSI for that vector cannot be sent. If software clears the mask bit and
the corresponding pending bit is set, the function must send the MSI request at that
time.

You should obtain the necessary MSI information (such as the message address and
data) from the configuration output interface (tl_cfg_*) to create the MWr TLP in
the format shown below to be sent via the Avalon-ST interface.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

73
 4. Interfaces
UG-20225 | 2021.02.18

Figure 36. Creating a MWr TLP for an MSI Request

MSI (Memory Write) Transaction

+0 +1 +2 +3

7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0
Fmt Type R T T E Attr AT 0 0 0 0Length
Byte 0 0 1 1 0 0 0 0 0 R TC R At tr H D P 00 0 0 0 0 0 1
Byte 4 Requester ID Tag Last DW First DW
0000 1111 Header
Byte 8 MSI Message Address [63:32]

Byte 12 MSI Message Address [31:0] 00

Byte 16 MSI Message Data 0000h Data

MSI Capability Structure

31 16 15 8 7 0
Message Control Next Capability Capability ID
Pointer (05h) DW0

Message Address [31:0] DW1

Message Address [63:32] DW2

Message Data DW3

Table 51. MSI Pending Bits Interface

Signal Name Direction Description Clock Domain EP/RP/BP

Function number select for the

coreclkout_
msi_pnd_func_i[2:0] I Pending Bits register in the MSI EP
capability structure.
hip

Byte select for Pending Bits

Register in the MSI Capability
Structure. For example if
msi_pnd_addr_i[1:0] = 00,
bits [7:0] of the Pending Bits
register will be updated with
coreclkout_
msi_pnd_addr_i[1:0] I EP
msi_pnd_byte_i[7:0]. If hip
msi_pnd_addr_i[1:0] = 01,
bits [15:8] of the Pending Bits
register will be updated with
msi_pnd_byte_i[7:0].

Indicate that function has a coreclkout_

msi_pnd_byte_i[7:0] I EP
pending associated message. hip

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

74
4. Interfaces
UG-20225 | 2021.02.18

The following figure shows the timings of msi_pnd_* signals in three scenarios. The
first scenario shows the case when the MSI pending bits register is not used. The
second scenario shows the case when only physical function 0 is enabled and the MSI
pending bits register is used. The last scenario shows the case when four physical
functions are enabled and the MSI pending bits register is used.

Figure 37. Example Timing Diagrams for msi_pnd* Signals

coreclkout_hip
msi_pnd_func_i[2:0] 0x0
msi_pnd_addr_i[1:0] 0x0
msi_pnd_byte_i[7:0] 0x0

coreclkout_hip
msi_pnd_func_i[2:0] 0x0
msi_pnd_addr_i[1:0] 0x0 0x1 0x2 0x3 0x0 0x1 0x2 0x3 0x0 0x1 0x2 0x3 0x0 0x1 0x2 0x3
msi_pnd_byte_i[7:0] B0 B1 B2 B3 B0 B1 B2 B3 B0 B1 B2 B3 B0 B1 B2 B3

coreclkout_hip
msi_pnd_func_i[2:0] 0x0 0x1 0x0 0x1
msi_pnd_addr_i[1:0] 0x0 0x1 0x2 0x3 0x0 0x1 0x2 0x3 0x0 0x1 0x2 0x3 0x0 0x1 0x2 0x3
msi_pnd_byte_i[7:0] B0 B1 B2 B3 B0 B1 B2 B3 B0 B1 B2 B3 B0 B1 B2 B3

There are 32 possible MSI messages. The number of messages requested by a

particular component does not necessarily correspond to the number of messages
allocated. For example, in the following figure, the Endpoint requests eight MSIs but is
only allocated two. In this case, you must design the Application Layer to use only two
allocated messages.

Figure 38. MSI Request Example

Root Complex

Endpoint Root CPU

Port

8 Requested
2 Allocated Interrupt
Block

Interrupt Register
The following table describes three example implementations. The first example
allocates all 32 MSI messages. The second and third examples only allocate 4
interrupts.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

75
 4. Interfaces
UG-20225 | 2021.02.18

Table 52. MSI Messages Requested, Allocated, and Mapped

MSI Allocated

32 4 4

System Error 31 3 3

Hot Plug and Power Management Event 30 2 3

Application Layer 29:0 1:0 2:0

MSI interrupts generated for Hot Plug, Power Management Events, and System Errors
always use Traffic Class 0. MSI interrupts generated by the Application Layer can use
any Traffic Class. For example, a DMA that generates an MSI at the end of a
transmission can use the same traffic control as was used to transfer data.

The following figure illustrates a possible implementation of the Interrupt Handler

Module with a per vector enable bit in the Application Layer. Alternatively, the
Application Layer could implement a global interrupt enable instead of this per vector
MSI.

Figure 39. Example Implementation of the Interrupt Handler Block

app_int_i

MSI info (from tl_cfg_ctl* /

Vector 0 Interrupt Enable 0 tl_cfg_addr* / tl_cfg_func*)
msi_req0 & Master Enable
R/W

Avalon-ST
IRQ Interrupt Request 0 single-dword MWR TLPs
Generation Arbitration &
TLP Generator msi_pnd_*
App Layer

Vector 1 Interrupt Enable 1

msi_req1
R/W

IRQ Interrupt Request 1

Generation

App Layer

Related Information
Handling PCIe Legacy and MSI Interrupts

4.6.3. MSI-X
The P-Tile IP for PCIe provides a Configuration Intercept Interface. User soft logic can
monitor this interface to get MSI-X Enable and MSI-X function mask related
information. User application logic needs to implement the MSI-X tables for all PFs and
VFs at the memory space pointed to by the BARs as a part of your Application Layer.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

76
4. Interfaces
UG-20225 | 2021.02.18

For more details on the MSI-X related information that you can obtain from the
Configuration Intercept Interface, refer to the MSI-X Registers section in the Registers
chapter.

MSI-X is an optional feature that allows the user application to support large amount
of vectors with independent message data and address for each vector.

When MSI-X is supported, you need to specify the size and the location (BARs and
offsets) of the MSI-X table and PBA. MSI-X can support up to 2048 vectors per
function versus 32 vectors per function for MSI.

A function is allowed to send MSI-X messages when MSI-X is enabled and the function
is not masked. The application uses the Configuration Output Interface (address 0x0C
bit[5:4]) or Configuration Intercept Interface to access this information.

When the application needs to generate an MSI-X, it will use the contents of the MSI-X
Table (Address and Data) and generate a Memory Write through the Avalon-ST
interface.

You can enable MSI-X interrupts by turning on the Enable MSI-X option under the
PCI Express/PCI Capabilities tab in the parameter editor. If you turn on the
Enable MSI-X option, you should implement the MSI-X table structures at the
memory space pointed to by the BARs as a part of your Application Layer.

The MSI-X Capability Structure contains information about the MSI-X Table and PBA
Structure. For example, it contains pointers to the bases of the MSI-X Table and PBA
Structure, expressed as offsets from the addresses in the function's BARs. The
Message Control register within the MSI-X Capability Structure also contains the MSI-X
Enable bit, the Function Mask bit, and the size of the MSI-X Table. For a picture of the
MSI-X Capability Structure, refer to Figure 87 on page 189.

MSI-X interrupts are standard Memory Writes, therefore Memory Write ordering rules
apply.

Example:

Table 53. MSI-X Configuration

MSI-X Vector MSI-X Upper Address MSI-X Lower Address MSI-X Data

0 0x00000001 0xAAAA0000 0x00000001

1 0x00000001 0xBBBB0000 0x00000002

2 0x00000001 0xCCCC0000 0x00000003

Table 54. PBA Table

PBA Table PBA Entries

Offset 0 0x0

If the application needs to generate an MSI-X interrupt (vector 1), it will read the MSI-
X Table information, generate a MWR TLP through the Avalon-ST interface and assert
the corresponding PBA bits (bit[1]) in a similar fashion as for MSI generation.

The generated TLP will be sent to address 0x00000001_BBBB0000 and the data will
be 0x00000002. When the MSI-X has been sent, the application can clear the
associated PBA bits.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

77
 4. Interfaces
UG-20225 | 2021.02.18

Related Information
Implementing MSI-X for PCI Express in Intel FPGA Devices

4.6.3.1. Implementing MSI-X Interrupts

Section 6.8.2 of the PCI Local Bus Specification describes the MSI-X capability and
table structures. The MSI-X capability structure points to the MSI-X Table structure
and MSI-X Pending Bit Array (PBA) registers. The BIOS sets up the starting address
offsets and BAR associated with the pointer to the starting address of the MSI-X Table
and PBA registers.

Figure 40. MSI-X Interrupt Components

PCIe with Avalon-ST I/F Application Layer
Host SW Programs Addr,
Data and Vector Control
RX RX Addr, Data MSI-X Table

Host
MSI-X PBA IRQ Source

TX Memory Write TX Memory Write TLP IRQ Monitor & Clr

TLP Processor

1. Host software sets up the MSI-X interrupts in the Application Layer by completing
the following steps:
a. Host software reads the Message Control register at 0x050 register to
determine the MSI-X Table size. The number of table entries is the <value
read> + 1.
The maximum table size is 2048 entries. Each 16-byte entry is divided in 4
fields as shown in the figure below. The MSI-X table can be accessed on any
BAR configured. The base address of the MSI-X table must be aligned to a
4 KB boundary.
b. The host sets up the MSI-X table. It programs MSI-X address, data, and
masks bits for each entry as shown in the figure below.

Figure 41. Format of MSI-X Table

DWORD 3 DWORD 2 DWORD 1 DWORD 0 Host Byte Addresses
Vector Control Message Data Message Upper Address Message Address Entry 0 Base
Vector Control Message Data Message Upper Address Message Address Entry 1 Base + 1 × 16
Vector Control Message Data Message Upper Address Message Address Entry 2 Base + 2 × 16

Vector Control Message Data Message Upper Address Message Address Entry (N - 1) Base + (N - 1) × 16

c. The host calculates the address of the <nth> entry using the following
formula:

nth_address = base address[BAR] + 16<n>

2. When Application Layer has an interrupt, it drives an interrupt request to the IRQ
Source module.
3. The IRQ Source sets appropriate bit in the MSI-X PBA table.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

78
4. Interfaces
UG-20225 | 2021.02.18

The PBA can use qword or dword accesses. For qword accesses, the IRQ Source
calculates the address of the <mth> bit using the following formulas:
qword address = <PBA base addr> + 8(floor(<m>/64))
qword bit = <m> mod 64

Figure 42. MSI-X PBA Table

Pending Bit Array (PBA) Address
Pending Bits 0 through 63 QWORD 0 Base
Pending Bits 64 through 127 QWORD 1 Base + 1 × 8

Pending Bits (( N - 1) div 64) × 64 through N - 1 QWORD (( N - 1) div 64) Base + ((N - 1) div 64) × 8
4. The IRQ Processor reads the entry in the MSI-X table.
a. If the interrupt is masked by the Vector_Control field of the MSI-X table,
the interrupt remains in the pending state.
b. If the interrupt is not masked, IRQ Processor sends Memory Write Request to
the TX slave interface. It uses the address and data from the MSI-X table. If
Message Upper Address = 0, the IRQ Processor creates a three-dword
header. If the Message Upper Address > 0, it creates a 4-dword header.
5. The host interrupt service routine detects the TLP as an interrupt and services it.

Related Information
• Floor and ceiling functions
• PCI Local Bus Specification, Rev. 3.0

4.7. Error Interface

This is an optional interface in the Intel P-Tile Avalon-ST IP for PCI Express that allows
the Application Layer to report errors to the IP core and vice versa. Specifically, the
Application Layer can report the different types of errors defined by the
app_error_info_i signal to the IP. For Advanced Error Reporting (AER), the
Application Layer can provide the information to log the TLP header and the error log
request via the app_err_* interface.

Note: The Intel P-Tile Avalon-ST IP for PCI Express enables the AER capability for Physical
Functions (PFs) by default. There is no AER implementation for Virtual Functions
(VFs). Use the VF Error Flag Interface instead of AER when using VFs.

Table 55. Error Interface Signals

Signal Name Direction Description Clock Domain EP/RP/BP

Indicates system error is

detected.
RP mode:
serr_out_o O A one-clock-cycle pulse on this coreclkout_hip EP/RP/BP
signal indicates if any device in
the hierarchy reports any of the
following errors and the
associated enable bit is set in
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

79
 4. Interfaces
UG-20225 | 2021.02.18

Signal Name Direction Description Clock Domain EP/RP/BP

the Root Control register:

ERR_COR, ERR_FATAL,
ERR_NONFATAL. Also asserted
when an internal error is
detected. The source of the error
will be logged in the Root Port
Error Status registers in the Port
Configuration and Status
registers.
EP mode:
Asserted when the P-Tile PCIe
Hard IP sends a message of
correctable/non-fatal/fatal error.
BP mode:
The transaction layer or data link
layer errors detected by the
Hard IP core trigger this signal.
Detailed information are logged
in the Bypass Mode Error Status
registers in the Port
Configuration and Status
registers.

Asserted when the Hard IP

enters the error mode. This
usually happens when the Hard
hip_enter_err_mod IP detects an uncorrectable RAM
O coreclkout_hip EP/RP/BP
e_o ECC error. Upon seeing the
assertion of this signal, you
should discard all the TLPs
received.

A one-cycle pulse on this signal

indicates that the data on
app_err_info, app_err_hdr,
app_err_valid_i I and app_err_func_num are coreclkout_hip EP/RP
valid in that cycle and
app_err_hdr_i will be valid
during the following four cycles.

This bus contains the header and

TLP prefix information for the
error TLP.
The 128-bit header and 32-bit
TLP prefix are sent to the Hard
IP over five cycles (32 bits of
app_err_hdr_i[31: information are sent in each
I coreclkout_hip EP/RP
0] clock cycle).
Cycle 1 : header[31:0]
Cycle 2 : header[63:32]
Cycle 3 : header[95:64]
Cycle 4 : header[127:96]
Cycle 5 : TLP prefix

This error bus carries the

following information:
• [0]: Malformed TLP
• [1]: Receiver overflow
app_err_info_i[12 • [2]: Unexpected completion
I coreclkout_hip EP/RP
:0]
• [3]: Completer abort
• [4]: Completion timeout
• [5]: Unsupported request
• [6]: Poisoned TLP received
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

80
4. Interfaces
UG-20225 | 2021.02.18

Signal Name Direction Description Clock Domain EP/RP/BP

• [7]: AtomicOp egress blocked

• [8]: Uncorrectable internal
error
• [9]: Correctable internal
error
• [10]: Advisory error
• [11]: TLP prefix blocked
• [12]: ACS violation

x16/x8:
app_err_func_num_ This bus contains the function
I number for the function that coreclkout_hip EP/RP
i[2:0]
asserts the error valid signal.
x4: NA

4.7.1. Completion Timeout Interface

The P-Tile IP for PCIe features a Completion timeout mechanism to keep track of Non-
Posted requests sent by the user application and the corresponding Completions
received. When the P-Tile IP detects a Completion timeout, it notifies the user
application by asserting the cpl_timeout_o signal.

When a Completion timeout happens, the user application can use the Avalon-MM
Completion Timeout Interface (for each port) to access the Completion timeout FIFO
in the Hard IP to get more detailed information about the event and update the AER
capability registers if required. After the completion timeout FIFO becomes empty, the
IP core deasserts the cpl_timeout_o signal.

The cpl_timeout_avmm interface is synchronized to the

cpl_timeout_avmm_clk_i clock.

Example:

When cpl_timeout_o is asserted, the user application can issue an Avalon-MM Read
to retrieve information from the Completion FIFO. Then, it can issue an Avalon-MM
Write to write 1 to bit[0] of the CONTROL register to get access to the next data.

Note: User application logic should not use this interface during TLP Bypass mode as the
information on this interface is not valid in that mode.

Table 56. Completion Timeout Interface

Signal Name Direction Description Clock domain EP/RP/BP

Indicates the event

that the completion
TLP for a request has
not been received
within the expected
time window.
The IP core asserts
cpl_timeout_o O this signal as long as coreclkout_hip EP/RP/BP
the completion
timeout FIFO in the
Hard IP is not empty.
You can obtain more
details about the
completion timeout
event by looking at
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

81
 4. Interfaces
UG-20225 | 2021.02.18

Signal Name Direction Description Clock domain EP/RP/BP

the signals on the

completion timeout
Avalon-MM interface
(listed below).

cpl_timeout_avmm_ Avalon-MM read cpl_timeout_avmm_cl

I EP/RP/BP
read_i enable. k_i

cpl_timeout_avmm_ Avalon-MM read data cpl_timeout_avmm_cl

O EP/RP/BP
readdata_o[7:0] outputs. k_i

This signal qualifies

the
cpl_timeout_avmm_ cpl_timeout_avmm_ cpl_timeout_avmm_cl
O EP/RP/BP
readdata_valid_o readdata_o signals k_i
into the Application
Layer.

cpl_timeout_avmm_ Avalon-MM write cpl_timeout_avmm_cl

I EP/RP/BP
write_i enable. k_i

cpl_timeout_avmm_ Avalon-MM write data cpl_timeout_avmm_cl

I EP/RP/BP
writedata_i[7:0] inputs. k_i

Avalon-MM address
inputs.
[20:3] : Reserved. Tie
them to 0.
cpl_timeout_avmm_ cpl_timeout_avmm_cl
I [2:0] : Address for the EP/RP/BP
addr_i[20:0] k_i
FIFO register. Refer to
the address map table
below for more
details.

When asserted, this

cpl_timeout_avmm_ signal indicates the IP cpl_timeout_avmm_cl
O EP/RP/BP
waitrequest_o core is not ready to k_i
take any request.

Avalon-MM clock.
50 MHz - 125 MHz
cpl_timeout_avmm_
I (Range) EP/RP/BP
clk_i
100 MHz
(Recommended)

Note: The Completion Timeout Interface has a separate address map that is isolated from
other address maps.

Table 57. Address Map for the Completion Timeout Interface

Address Name Access Type Description

0x0 STATUS RO [7:2] : Reserved

[1] : Completion timeout
FIFO full
[0] : Completion timeout
FIFO empty

0x1 CONTROL WO [7:1] : Reserved

[0] : Read (popping data
from the FIFO). You need to
read all the information
regarding the timed out
request before writing 1 to
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

82
4. Interfaces
UG-20225 | 2021.02.18

Address Name Access Type Description

bit 0 of the CONTROL

register. Writing to bit 0 of
the CONTROL register
makes the next data appear.

0x2 VF RO [7:0] : vfunc_num[7:0]

Virtual Function number for
the VF that initiates the non-
posted transaction for which
the completion timeout is
observed.

0x3 PF RO [7] : vfunc_active

[6] : Reserved
[5:3] : func_num[2:0]
Physical function number
(least significant 8 bits) for
the PF that initiates the non-
posted transaction for which
the completion timeout is
observed.
[2:0] : vfunc_num[10:8]
Virtual Function number
(most significant 3 bits) for
the VF that initiates the
memory read request for
which the completion
timeout is observed.

0x4 LEN1 RO [7:0] : cpl_lenn[7:0]

Transfer length in bytes
(least significant 8 bits), of
the expected completion
that timed out for the non-
posted transaction. For a
split completion, it indicates
the number of bytes
remaining to be delivered
when the completion timed
out (Max length is Max Read
request size. Ex: 4K Bytes =
2^12 bytes)

0x5 LEN2 RO [7:4] : Reserved

[3:0] : cpl_lenn[11:8]
Transfer length in bytes
(most significant 4 bits), of
the expected completion
that timed out for the non-
posted transaction. For a
split completion, it indicates
the number of bytes
remaining to be delivered
when the completion timed
out (Max length is Max Read
request size. Ex: 4K Bytes =
2^12 bytes)

0x6 TAG1 RO [7:0] : cpl_tag[7:0]

Tag ID (least significant 8
bits) of the expected
completion that timed out
for the non-posted
transaction.

0x7 TAG2 RO [7:5] : cpl_tc[2:0]

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

83
 4. Interfaces
UG-20225 | 2021.02.18

Address Name Access Type Description

Traffic class of the expected

completion that timed out
for the non-posted
transaction.
[4:3] : cpl_attr[1:0]
Attribute of the expected
completion that timed out
for the non-posted
transaction. ID based
ordering is not supported.
[4] -> Relaxed ordering, [3]
-> No Snoop
[2] : Reserved
[1:0]: cpl_tag[9:8]
Tag ID (most significant 2
bits) of the expected
completion that timed out
for the non-posted
transaction.

4.8. Hot Plug Interface (RP Only)

Hot Plug support means that the device can be added to or removed from a system
during runtime. The Hot Plug Interface in the P-Tile IP for PCIe allows an Intel FPGA
with this IP to safely provide this capability.

This section describes the signals reported by the on-board hot plug components in
the Downstream Port. This interface is available only if the Slot Status Register
of the PCI Express Capability Structure is enabled.

Refer to the Slot Status Register of the PCI Express Capability

Structure for additional information.

Table 58. Hot Plug Interface

Signal Name Direction Description Clock Domain EP/RP/BP

sys_atten_button_pressed_i I Attention Button coreclkout_hip RP

Pressed. Indicates
that the system
attention button was
pressed, and sets
the Attention Button
Pressed bit in the
Slot Status
Register.

sys_pwr_fault_det_i I Power Fault coreclkout_hip RP

Detected. Indicates
the power controller
detected a power
fault at this slot.

sys_mrl_sensor_chged_i I MRL Sensor coreclkout_hip RP

Changed. Indicates
that the state of the
MRL sensor has
changed.
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

84
4. Interfaces
UG-20225 | 2021.02.18

Signal Name Direction Description Clock Domain EP/RP/BP

sys_pre_det_chged_i I Presence Detect coreclkout_hip RP

Changed. Indicates
that the state of the
card presence
detector has
changed.

sys_cmd_cpled_int_i I Command coreclkout_hip RP

Completed Interrupt.
Indicates that the
Hot Plug controller
completed a
command.

sys_pre_det_state_i I Indicates whether or coreclkout_hip RP

not a card is present
in the slot.
0 : slot is empty.
1 : card is present in
the slot.

sys_mrl_sensor_state_i I MRL Sensor State. coreclkout_hip RP

Indicates the state
of the manually
operated retention
latch (MRL) sensor.
0 : MRL is closed.
1 : MRL is open.

sys_eml_interlock_engaged_ I Indicates whether coreclkout_hip RP

i the system
electromechanical
interlock is engaged,
and controls the
state of the
electromechanical
interlock status bit in
the Slot Status
Register.

sys_aux_pwr_det_i I Auxiliary Power coreclkout_hip RP

Detected. Used to
report to the host
software that
auxiliary power
(Vaux) is present.
Refer to the Device
Status Register
in the PCI Express
Capability
Structure.

4.9. Power Management Interface

Software programs the device into a D-state by writing to the Power Management
Control and Status register in the PCI Power Management Capability
Structure. The power management output signals indicate the current power state.
The IP core supports the two mandatory power states: D0 (full power) and D3
(preparation for a loss of power). It does not support the optional D1 and D2 low-
power states.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

85
 4. Interfaces
UG-20225 | 2021.02.18

The correspondence between the device power states (D states) and link power states
(L states) is as follows:

Table 59. Relationship Between Device and Link Power States

Device Power State Link Power State

D0 L0

D1 (not supported) L1

D2 (not supported) L1

D3 L1, L2/L3 Ready

P-Tile does not support ASPM.

Table 60. Power Management Interface

Signal Name Direction Description Clock Domain EP/RP/BP

Indicates the current power

state.
• 3'b000 : L0 or IDLE
pm_state_o[2:0] O • 3'b001 : L0s coreclkout_hip EP/RP/BP
• 3'b010 : L1
• 3'b011 : L2
• 3'b100 : L3

Power management D-state

for each function.
x16/x8: • 4'b0001 : D0
pm_dstate_o[31:0] • 4'b0010 : D1
O Async EP/RP/BP
x4: • 4'b0100 : D2
pm_dstate_o[3:0] • 4'b1000 : D3
• 4'b0000 : uninitialized or
invalid

The application logic asserts

this signal for one cycle to
x16/x8: wake up the Power
apps_pm_xmt_pme_ Management Capability (PMC)
I coreclkout_hip EP/BP
i[7:0] state machine from a D1, D2,
x4: NA or D3 power state. Upon
wake-up, the IP core sends a
PM_PME message.

This signal is a request from

the Application Layer to
generate a PM_Turn_Off
message. The Application
Layer must assert this signal
for one clock cycle. The IP core
apps_pm_xmt_turno
I does not return an coreclkout_hip RP
ff_i acknowledgement or grant
signal. The Application Layer
must not pulse the same
signal again until the previous
message has been
transmitted.

The Application Layer uses this

app_init_rst_i I signal to request a hot reset to coreclkout_hip RP
downstream devices. The hot
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

86
4. Interfaces
UG-20225 | 2021.02.18

Signal Name Direction Description Clock Domain EP/RP/BP

reset request will be sent when

a single-cycle pulse is applied
to this pin.

When this signal is asserted,

the PCIe Controller responds
to Configuration TLPs with a
CRS (Configuration Retry
Status) if it has not already
app_req_retry_en_
I responded to a Configuration coreclkout_hip EP/BP
i TLP with non-CRS status since
the last reset. You can use this
signal to hold off on
enumeration. This input is not
used for Root Ports.

4.10. Configuration Output Interface

The Transaction Layer configuration output (tl_cfg) bus provides a subset of the
information stored in the Configuration Space. Use this information in conjunction with
the app_err* signals to understand TLP transmission problems.

Note: User application logic should not use this interface during TLP Bypass mode as the
information on this interface is not valid in that mode.

Table 61. Configuration Output Interface

Signal Name Direction Description Clock Domain EP/RP/BP

Multiplexed data output from the

register specified by
tl_cfg_ctl_o[15:0 tl_cfg_add_o[4:0]. The
O coreclkout_hip EP/RP/BP
] detailed information for each field
in this bus is defined in the
following table.

This address bus contains the

index indicating which
tl_cfg_add_o[4:0] O Configuration Space register coreclkout_hip EP/RP/BP
information is being driven onto
the tl_cfg_ctl_o[15:0] bits.

Specifies the function whose

Configuration Space register values
x16/x8: are being driven out on
tl_cfg_func_o[2:0 tl_cfg_ctl_o[15:0].
O coreclkout_hip EP/RP/BP
] • 3'b000: Physical Function 0
x4: NA (PF0)
• 3'b001: PF1
and so on

The table below provides the tl_cfg_add_o[4:0] to tl_cfg_ctl_o[15:0]

mapping.

Table 62. Multiplexed Configuration Information Available on tl_cfg_ctl

tl_cfg_add_o[4:0] tl_cfg_ctl_o[15:8] tl_cfg_ctl_o[7:0]

[15]: memory space enable Device control:

5'h00
[14]: IDO completion enable [7]: bus master enable
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

87
 4. Interfaces
UG-20225 | 2021.02.18

tl_cfg_add_o[4:0] tl_cfg_ctl_o[15:8] tl_cfg_ctl_o[7:0]

[13]: perr_en
[12]: serr_en
[6]: extended tag enable
[11]: fatal_err_rpt_en
[5:3]: maximum read request size
[10]: nonfatal_err_rpt_en
[2:0]: maximum payload size
[9]: corr_err_rpt_en
[8]: unsupported_req_rpt_en

[15]: IDO request enable

[14]: No Snoop enable
5'h01 bus number
[13]: Relaxed Ordering enable
[12:8]: Device number

[15]: pm_no_soft_rst [7:5]: reserved

[14]: RCB control [4]: system power control
5'h02 [13]: Interrupt Request (IRQ) disable [3:2]: system attention indicator
[12:8]: PCIe Capability IRQ message control
number [1:0]: system power indicator control

5'h03 Number of VFs [15:0]

[15]: reserved
[7]: ARI forward enable
[14]: AtomicOP Egress Block field
[6]: Atomic request enable
(cfg_atomic_egress_block)
5'h04 [5:3]: TPH ST mode
[13:9]: ATS Smallest Translation Unit
[2:1]: TPH enable
(STU)[4:0]
[0]: VF enable
[8]: ATS cache enable

[15:12]: auto negotiation link speed.

Link speed encoding values are:
• Gen1 : 0x1
• Gen2 : 0x2
5'h05
• Gen3 : 0x4
• Gen4 : 0x8
[11:1]: Index of Start VF [10:0]
[0]: reserved

5'h06 MSI Address [15:0]

5'h07 MSI Address [31:16]

5'h08 MSI Address [47:32]

5'h09 MSI Address [63:48]

5'h0A MSI Mask [15:0]

5'h0B MSI Mask [31:16]

[7]: Enable extended message data for

MSI (cfg_msi_ext_data_en)
[15]: cfg_send_f_err
[6]: MSI-X func mask
[14]: cfg_send_nf_err
5'h0C [5]: MSI-X enable
[13]: cfg_send_cor_err
[4:2]: Multiple MSI enable
[12:8]: AER IRQ message number [1]: 64-bit MSI
[0]: MSI enable

5'h0D MSI Data [15:0]

5'h0E AER uncorrectable error mask [15:0]

5'h0F AER uncorrectable error mask [31:16]

continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

88
4. Interfaces
UG-20225 | 2021.02.18

tl_cfg_add_o[4:0] tl_cfg_ctl_o[15:8] tl_cfg_ctl_o[7:0]

5'h10 AER correctable error mask [15:0]

5'h11 AER correctable error mask [31:16]

5'h12 AER uncorrectable error severity [15:0]

5'h13 AER uncorrectable error severity [31:16]

[7]: ACS function group enable

(cfg_acs_func_grp_en)
[6]: ACS direct translated P2P enable
(cfg_acs_p2p_direct_tranl_en)
[5]: ACS P2P egress control enable
(cfg_acs_egress_ctrl_en)
[4]: ACS upstream forwarding enable
(cfg_acs_up_forward_en)
[15:8]: ACS Egress Control Register [3]: ACS P2P completion redirect
5'h14
(cfg_acs_egress_ctrl_vec) enable
(cfg_acs_p2p_compl_redirect_en
)
[2]: ACS P2P request redirect enable
(cfg_acs_p2p_req_redirect_en)
[1]: ACS translation blocking enable
(cfg_acs_at_blocking_en)
[0]: ACS source validation enable (RP)
(cfg_acs_validation_en)

[15]: reserved
[14]: 10-bit tag requester enable
(cfg_10b_tag_req_en)
[13]: VF 10-bit tag requester enable
(cfg_vf_10b_tag_req_en)
[12]: PRS_RESP_FAILURE [7:3]: reserved
5'h15 (cfg_prs_response_failure) [2:0]: ARI function group
(cfg_ari_func_grp)
[11]: PRS_UPRGI (cfg_prs_uprgi)
[10]: PRS_STOPPED
(cfg_prs_stopped)
[9]: PRS_RESET (cfg_prs_reset)
[8]: PRS_ENABLE (cfg_prs_enable)

PRS_OUTSTANDING_ALLOCATION
5'h16 (cfg_prs_outstanding_allocatio
n) [15:0]

PRS_OUTSTANDING_ALLOCATION
5'h17 (cfg_prs_outstanding_allocatio
n) [31:16]

[7]: Infinite credits for Posted header

[6]: Infinite credits for Posted data
[5]: Infinite credits for Completion
[15:10]: reserved header
[9]: Disable autonomous generation of [4]: Infinite credits for Completion data
LTR clear message [3]: End-end TLP prefix blocking
5'h18
(cfg_disable_ltr_clr_msg) (cfg_end2end_tlp_pfx_blck)
[8]: LTR mechanism enable [2]: PASID enable
(cfg_ltr_m_en) (cfg_pf_pasid_en)
[1]: Execute permission enable
(cfg_pf_passid_execute_perm_en
)
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

89
 4. Interfaces
UG-20225 | 2021.02.18

tl_cfg_add_o[4:0] tl_cfg_ctl_o[15:8] tl_cfg_ctl_o[7:0]

[0]: Privileged mode enable

(cfg_pf_passid_priv_mode_en)

[7]: Slot control power fault detect

enable (cfg_pwr_fault_det_en)
[6]: Slot control MRL sensor changed
enable (cfg_mrl_sensor_chged_en)
[5]: Slot control presence detect
changed enable
(cfg_pre_det_chged_en)
[15:9]: reserved [4]: Slot control hot plug interrupt
[8]: Slot control attention button enable (cfg_hp_int_en)
5'h19
pressed enable [3]: Slot control command completed
(cfg_atten_button_pressed_en) interrupt enable
(cfg_cmd_cpled_int_en)
[2]: Slot control DLL state change
enable (cfg_dll_state_change_en)
[1]: Slot control accessed
(cfg_hp_slot_ctrl_access)
[0]: PF’s SERR# enable
(cfg_br_ctrl_serren)

LTR maximum snoop latency register

5'h1A
(cfg_ltr_max_latency[15:0])

LTR maximum no-snoop latency

5'h1B register
(cfg_ltr_max_latency[31:16])

[5:0]: auto negotiation link width

6’h01 = x1
[15:8]: enabled Traffic Classes (TCs) 6’h02 = x2
5'h1C
(cfg_tc_enable[7:0]) 6’h04 = x4
6’h08 = x8
6’h10 = x16

5'h1D MSI Data[31:16]

5'h1E N/A

5'h1F N/A

Note: The information on the Configuration Output (tl_cfg) bus is time-division

multiplexed (TDM).

• When tl_cfg_func[2:0] = 3'b000, tl_cfg_ctl[31:0] drive out the PF0

Configuration Space register values.
• Then, tl_cfg_func[2:0] are incremented to 3'b001.
• When tl_cfg_func[2:0] = 3'b001, tl_cfg_ctl[31:0] drive out the PF1
Configuration Space register values.
• This pattern repeats to cover all enabled PFs.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

90
4. Interfaces
UG-20225 | 2021.02.18

Figure 43. Configuration Output Interface Timing Diagram

coreclkout_hip

tl_cfg_add_0[4:0] 0x00 0x01 0x02 0x03 0x00 0x01 0x02

tl_cfg_ctl_0[15:0] PF0 DATA0 PF0 DATA1 PF0 DATA2 PF0 DATA3 PF1 DATA0 PF1 DATA1 PF1 DATA2

tl_cfg_func_0[2:0] 0 1

Note: The P-Tile IP for PCIe provides a data link layer timer update output. Details on this
signal are in the table below. When this signal asserts, you can sample the
tl_cfg_ctl_o bus to see the new link speed, link width or max payload size and
update the Replay/Ack-Nak timers accordingly.

Table 63. Data Link Layer Timer Update Signal

Signal Name Direction Description Clock Domain EP/RP/BP

dl_timer_update_o O Active high pulse that coreclkout_hip EP/RP/BP

asserts whenever the
current link speed,
link width, or max
payload size changes.
When any of these
parameters changes,
the IP's internal
Replay/Ack-Nak timers
default back to their
internally calculated
PCIe tables.
To override these
default values,
reprogram the Port
Logic register when
these events occur.

4.11. Configuration Intercept Interface (EP Only)

The Configuration Intercept Interface (CII) allows the application logic to detect the
occurrence of a Configuration (CFG) request on the link and to modify its behavior.

The application logic should detect the CFG request at the rising edge of cii_req.
Due to the latency of the EMIB, the cii_req can be deasserted many cycles after the
deassertion of cii_halt.

The application logic can use the CII to:

• Delay the processing of a CFG request by the controller. This allows the application
to perform any housekeeping task first.
• Overwrite the data payload of a CfgWr request. The application logic can also
overwrite the data payload of a CfgRd completion TLP.

This interface also allows you to implement the Intel Vendor Specific Extended
Capability (VSEC) registers. All configuration accesses targeting the Intel VSEC
registers (addresses 0xD00 to 0xFFF) are automatically mapped to this interface and
can be monitored via this interface.

If you are not using this interface, tie cii_halt_p0/1 to logic 0.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

91
 4. Interfaces
UG-20225 | 2021.02.18

Figure 44. Configuration Intercept Interface Timing Diagram

coreclkout_hip

cii_req_o

cii_* valid

cii_halt_i

cii_override_en_i valid

cii_override_din_i[31:0] valid

Table 64. Configuration Intercept Interface

Signal Name Direction Description Clock domain EP/RP/BP

cii_req_o O Indicates the CFG request is

intercepted and all the other CII coreclkout_hip EP
signals are valid.

cii_hdr_poisoned_o O The poisoned bit in the received

coreclkout_hip EP
TLP header on the CII.

cii_hdr_first_be_ O The first dword byte enable field

o[3:0] in the received TLP header on the coreclkout_hip EP
CII.

cii_func_num_o[2:0 O The function number in the

received TLP header on the CII. coreclkout_hip EP
]

cii_wr_o O Indicates that cii_dout_p0/1 is

valid. This signal is asserted only coreclkout_hip EP
for a configuration write request.

cii_addr_o[9:0] O The double word register address

in the received TLP header on the coreclkout_hip EP
CII.

cii_dout_o[31:0] O Received TLP payload data from

the link partner to your application
client. The data is in little endian coreclkout_hip EP
format. The first received payload
byte is in [7:0].

cii_override_en_i I Override enable. When the

application logic asserts this input,
the PCIe Hard IP overrides the
CfgWr payload or CfgRd coreclkout_hip EP
completion using the data
supplied by the application logic
on cii_override_din.

cii_override_din_ I Override data.

coreclkout_hip EP
i[31:0]
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

92
4. Interfaces
UG-20225 | 2021.02.18

Signal Name Direction Description Clock domain EP/RP/BP

• CfgWr: override the write data

to the PCIe Hard IP register
with data supplied by the
application logic on
cii_override_din.
• CfgRd: override the data
payload of the completion TLP
with data supplied by the
application logic on
cii_override_din.

cii_halt_i I Flow control input signal. When

cii_halt_p0/1 is asserted, the
PCIe Hard IP halts the processing coreclkout_hip EP
of CFG requests for the PCIe
configuration space registers.

4.12. Hard IP Reconfiguration Interface

The Hard IP reconfiguration interface is an Avalon-MM slave interface with a 21-bit
address and an 8-bit data bus. It is also sometimes referred to as the User Avalon-MM
Interface. You can use this interface to dynamically modify the value of configuration
registers. Note that after a warm reset or cold reset, changes made to the
configuration registers of the Hard IP via the Hard IP reconfiguration interface are lost
as these registers revert back to their default values.

Note: This interface can be used in Endpoint, Root Port and TLP Bypass modes. However, it
must be enabled if Root Port or TLP Bypass mode is selected.

In Root Port mode, the application logic uses the Hard IP reconfiguration interface to
access its PCIe configuration space to perform link control functions (such as Hot
Reset, link disable, or link retrain).

In TLP Bypass mode, the Hard IP forwards the received Type0/1 Configuration request
TLPs to the application logic, which must respond with Completion TLPs with a status
of Successful Completion (SC), Unsupported Request (UR), Configuration Request
Retry Status (CRS), or Completer Abort (CA). If a received Configuration request TLP
needs to update a PCIe configuration space register, the application logic needs to use
the Hard IP reconfiguration interface to access that PCIe configuration space register.

Table 65. Hard IP Reconfiguration Interface

Signal Name Direction Description Clock Domain EP/RP/BP

Reconfiguration
clock
50 MHz - 125 MHz
hip_reconfig_clk I EP/RP/BP
(Range)
100 MHz
(Recommended)

Avalon-MM read hip_reconfig_cl

hip_reconfig_readdata_o[7:0] O EP/RP/BP
data outputs k

Avalon-MM read
data valid. When hip_reconfig_cl
hip_reconfig_readdatavalid_o O EP/RP/BP
asserted, the data k
on
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

93
 4. Interfaces
UG-20225 | 2021.02.18

Signal Name Direction Description Clock Domain EP/RP/BP

hip_reconfig_re
addata_o[7:0] is
valid.

Avalon-MM write hip_reconfig_cl

hip_reconfig_write_i I EP/RP/BP
enable k

Avalon-MM read hip_reconfig_cl

hip_reconfig_read_i I EP/RP/BP
enable k

hip_reconfig_cl
hip_reconfig_address_i[20:0] I Avalon-MM address EP/RP/BP
k

Avalon-MM write hip_reconfig_cl

hip_reconfig_writedata_i[7:0] I EP/RP/BP
data inputs k

When asserted, this

signal indicates that
hip_reconfig_cl
hip_reconfig_waitrequest_o O the IP core is not EP/RP/BP
ready to respond to
k
a request.

Reset signal. You

can tie it to ground
or leave it floating
dummy_user_avmm_rst I EP/RP/BP
when using the Hard
IP Reconfiguration
Interface.

Reading and Writing to the Hard IP Reconfiguration Interface

Reading from the Hard IP reconfiguration interface of the P-Tile Avalon-ST IP for PCI
Express retrieves the current value at a specific address. Writing to the reconfiguration
interface changes the data value at a specific address. Intel recommends that you
perform read-modify-writes when writing to a register, because two or more features
may share the same reconfiguration address.

Modifying the PCIe configuration registers directly affects the behavior of the PCIe
device.

Figure 45. Timing Diagram to Perform Read and Write Operations Using the Hard IP
Reconfiguration Interface
hip_reconfig_clk

hip_reconfig_addr_i[20:0] 0x0003E8 0x0003E9

hip_reconfig_write_i

hip_reconfig_writedata_i[7:0] 0x01

hip_reconfig_read_i

hip_reconfig_readdatavalid

hip_reconfig_readdata_o[7:0] 0x01

hip_reconfig_waitrequest_o

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

94
4. Interfaces
UG-20225 | 2021.02.18

4.12.1. Address Map for the User Avalon-MM Interface

The User Avalon-MM interface provides access to the configuration registers and the IP
core registers. This interface includes an 8-bit data bus and a 21-bit address bus
(which contains the byte addresses).

There are two methods to access the configuration registers:

• Using direct User Avalon-MM interface (byte access)
• Using the Debug (DBI) register access (dword access). This method is useful when
you need to read/write the entire 32 bits at one time (Counter/ Lane Margining,
etc.)

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

95
 4. Interfaces
UG-20225 | 2021.02.18

Figure 46. Address Map for the User Avalon-MM Interface

0x1fffff 0x1fffff

0x104200 Debug_DBI_Data/Debug_DBI_Addr 0x104200 Debug_DBI_Data/Debug_DBI_Addr

0x104068 User Avalon-MM Port Configuration Register 0x104068 User Avalon-MM Port Configuration Register

0x008000

0x007000 PF7 PCie Configuration Registers

PF6 PCie Configuration Registers

0x006000
PF5 PCie Configuration Registers
0x005000
PF4 PCie Configuration Registers
0x004000
PF3 PCie Configuration Registers
0x003000
PF2 PCie Configuration Registers
0x002000
PF1 PCie Configuration Registers
0x001000 0x001000
PF0 PCie Configuration Registers PF0 PCie Configuration Registers
0x000000 0x000000
(a) hip_core_x16/x8 user avmm (b) hip_core_x4 user avmm
address space address space

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

96
4. Interfaces
UG-20225 | 2021.02.18

Table 66. Configuration Space Offsets

Registers User Avalon-MM Offsets Comments

Physical function 0 0x0000 Refer to Appendix A for more details of

the PF configuration space. This PF is
available for x16, x8 and x4 cores.

Physical function 1 0x1000 Refer to Appendix A for more details of

the PF configuration space. This PF is
available for x16 and x8 cores only.

Physical function 2 0x2000 Refer to Appendix A for more details of

the PF configuration space. This PF is
available for x16 and x8 cores only.

Physical function 3 0x3000 Refer to Appendix A for more details of

the PF configuration space. This PF is
available for x16 and x8 cores only.

Physical function 4 0x4000 Refer to Appendix A for more details of

the PF configuration space. This PF is
available for x16 and x8 cores only.

Physical function 5 0x5000 Refer to Appendix A for more details of

the PF configuration space. This PF is
available for x16 and x8 cores only.

Physical function 6 0x6000 Refer to Appendix A for more details of

the PF configuration space. This PF is
available for x16 and x8 cores only.

Physical function 7 0x7000 Refer to Appendix A for more details of

the PF configuration space. This PF is
available for x16 and x8 cores only.

User Avalon-MM Port Configuration 0x104068 Refer to User Avalon-MM Port

Register Configuration Register (Offset
0x104068) on page 97 for more
details.

Debug (DBI) Register 0x104200 to 0x104204 Refer to Using the Debug Register
Interface Access (Dword Access) on
page 99 for more details.

Note: The x4 configuration only supports the RP mode. Therefore, this configuration does
not support the multi-function feature.

4.12.1.1. User Avalon-MM Port Configuration Register (Offset 0x104068)

Table 67. User Avalon-MM Port Configuration Register

Bits Register Description Default Value Access

[31:29] Reserved 0x0 RO

[28:18] Select the virtual function 0x0 RW/RO

number.

[17] To access the virtual function 0x0 RW/RO

registers, this bit should be
set to one.
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

97
 4. Interfaces
UG-20225 | 2021.02.18

Bits Register Description Default Value Access

[16:2] Reserved 0x0 RO

[1] Reserved. Clear this bit for 0x0 RW/RO

access to standard PCIe
configuration registers.

[0] If set, it allows access to 0x0 RW/RO

Intel VSEC registers.

4.12.2. Configuration Registers Access

4.12.2.1. Using Direct User Avalon-MM Interface (Byte Access)

Targeting PF Configuration Space Registers

User application needs to specify the offsets of the targeted PF registers.

For example, if the application wants to read the MSI Capability Register of PF0, it will
issue a Read with address 0x0050 to target the MSI Capability Structure of PF0.

Figure 47. PF Configuration Space Registers Access Timing Diagram

HIP reconfig clk

HIP reconfig addr 0x000050

HIP reconfig read

HIP reconfig readdata PF0 MSI Cap

HIP reconfig readdatavalid

Targeting VF Configuration Space Registers

User application needs to first specify the VF number of the targeted configuration
register.

The application needs to program the User Avalon-MM Port Configuration Register at
offset 0x104068 accordingly.

For example, to read VF3's MSI-X Capability registers, the user application needs to:
1. Issue a user Avalon-MM Write request with address 0x104068 and data 0xE
( vf_num[28:18] = 3, vf _select[17] = 1, vsec[0]=0).
2. Issue a user Avalon-MM Read request with address 0xB0 to access VF3 registers.

Note: You need to reprogram the Port Configuration and Status Register to access PF
registers.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

98
4. Interfaces
UG-20225 | 2021.02.18

Figure 48. VF Configuration Space Registers Access Timing Diagram

HIP reconfig clk

HIP reconfig addr 0x104068 0xB0

HIP reconfig write

HIP reconfig writedata 0x0E

HIP reconfig read

HIP reconfig readdata VF3 MSIX Table

HIP reconfig readdatavalid

Targeting VSEC Registers

User application needs to program the VSEC field (0x104068 bit[0]) first. Then all
accesses from the user Avalon-MM interface starting at offset 0xD00 will be translated
to VSEC configuration space registers.

Figure 49. VSEC Registers Access Timing Diagram

HIP reconfig clk

HIP reconfig addr 0x104068 0xD00

HIP reconfig write

HIP reconfig writedata 0x01

HIP reconfig read

HIP reconfig waitrequest

HIP reconfig readdata VSEC Cap

HIP reconfig readdatavalid

4.12.2.2. Using the Debug Register Interface Access (Dword Access)

DEBUG_DBI_ADDR register is located at user Avalon-MM offsets 0x104204 to

0x104207 (corresponding to byte 0 to byte 3).

Table 68. DEBUG_DBI_ADDR Register

Names Bits R/W Descriptions

d_done 31 RO 1: indicates debug DBI read/

write access done

d_write 30 R/W 1: write access

0: read access

d_warm_reset 29 RO 1: normal operation

0: warm reset is on-going

d_vf 28:18 R/W Specify the virtual function

number.

d_vf_select 17 R/W To access the virtual function

registers, set this bit to one.

d_pf 16:14 R/W Specify the physical function

number.

reserved 13:12 R/W Reserved

continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

99
 4. Interfaces
UG-20225 | 2021.02.18

Names Bits R/W Descriptions

d_addr 11:2 R/W Specify the DW address for

the P-Tile Hard IP DBI
interface.

d_shadow_select 1 R/W Reserved. Clear this bit for

access to standard PCIe
configuration registers.

d_vsec_select 0 R/W If set, this bit allows access

to Intel VSEC registers.

DEBUG_DBI_DATA register is located at user Avalon-MM offsets 0x104200 to

0x104203 (corresponding to byte 0 to byte 3).

Table 69. DEBUG_DBI_DATA Register

Names Bits R/W Descriptions

d_data 31:0 R/W Read or write data for the P-

Tile Hard IP register access.

To write all 32 bits in a Debug register at a time:

1. Use the user_avmm interface to access 0x104200 to 0x104203 to write the data
first.
2. Use the user_avmm interface to access 0x104204 to 0x104206 to set the address
and control bits.
3. Use the user_avmm interface to write to 0x104207 to enable the read/write bit
(bit[30]).
4. Use the user_avmm interface to access 0x104207 bit[31] to poll if the write is
complete.

Figure 50. DBI Register Write Timing Diagram

HIP reconfig clk

HIP reconfig addr 0x104200 0x104201 0x104202 0x104203 0x104204 0x104205 0x104206 0x104207

HIP reconfig write

HIP reconfig writedata 0x01 0x23 0x45 0x67 ADDR CTRL 0x4

HIP reconfig waitrequest

HIP reconfig read

HIP reconfig readdata 0x4 0xC

HIP reconfig readdatavalid

To read all 32 bits in a Debug register at a time:

1. Use the user_avmm interface to access 0x104204 to 0x104206 to set the address
and control bits.
2. Use the user_avmm interface to write to 0x104207 to enable the read bit
(bit[30]).
3. Use the user_avmm interface to access 0x104207 bit[31] to poll if the read is
complete.
4. Use the user_avmm interface to access 0x104200 to 0x104203 to read the data

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

100
4. Interfaces
UG-20225 | 2021.02.18

Figure 51. DBI Register Read Timing Diagram

HIP reconfig clk

HIP reconfig addr 0x104204 0x104205 0x104206 0x104207 0x104207 0x104200 0x104201 0x104202 0x104203

HIP reconfig write

HIP reconfig writedata ADDR CTRL 0x0

HIP reconfig waitrequest

HIP reconfig read

HIP reconfig readdata 0x0 0x8 D0 D1 D2 D3

HIP reconfig readdatavalid

4.13. PHY Reconfiguration Interface

The PHY reconfiguration interface is an optional Avalon-MM slave interface with a
26-bit address and an 8-bit data bus. Use this bus to read the value of PHY registers.
Refer to Table 110 on page 160 for details on addresses and bit mappings for the PHY
registers that you can access using this interface.

These signals are present when you turn on Enable PHY reconfiguration on the
Top-Level Settings tab using the parameter editor.

Please note that the PHY reconfiguration interface is shared among all the PMA quads.

Table 70. PHY Reconfiguration Interface

Signal Name Direction Description Clock Domain EP/RP/BP

Reconfiguration clock
xcvr_reconfig_clk I 50 MHz - 125 MHz (Range) EP/RP/BP
100 MHz (Recommended)

xcvr_reconfig_readdata[7:0 Avalon-MM read data xcvr_reconfig_

O EP/RP/BP
] outputs clk

Avalon-MM read data valid.

xcvr_reconfig_readdatavali When asserted, the data on xcvr_reconfig_
O EP/RP/BP
d xcvr_reconfig_readdat clk
a[7:0] is valid.

xcvr_reconfig_
xcvr_reconfig_write I Avalon-MM write enable EP/RP/BP
clk

Avalon-MM read enable.

This interface is not
pipelined. You must wait for
the return of the xcvr_reconfig_
xcvr_reconfig_read I EP/RP/BP
xcvr_reconfig_readdat clk
a[7:0] from the current
read before starting another
read operation.

Avalon-MM address
[25:21] are used to indicate
the Quad.
xcvr_reconfig_address[25:0 5'b00001 : Quad 0 xcvr_reconfig_
I EP/RP/BP
] clk
5'b00010 : Quad 1
5'b00100 : Quad 2
5'b01000 : Quad 3
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

101
 4. Interfaces
UG-20225 | 2021.02.18

Signal Name Direction Description Clock Domain EP/RP/BP

[20:0] are used to indicate

the offset address.

xcvr_reconfig_writedata[7: xcvr_reconfig_
I Avalon-MM write data inputs EP/RP/BP
0] clk

When asserted, this signal

indicates that the PHY is not xcvr_reconfig_
xcvr_reconfig_waitrequest O EP/RP/BP
ready to respond to a clk
request.

Reading from the PHY Reconfiguration Interface

Reading from the PHY reconfiguration interface of the P-Tile Avalon-ST IP for PCI
Express retrieves the current value at a specific address.

Figure 52. Timing Diagram to Perform Read Operations Using the PHY Reconfiguration
Interface
xcvr_reconfig_clk

xcvr_reconfig_address 0x000006

xcvr_reconfig_read

xcvr_reconfig_readdatavalid

xcvr_reconfig_readdata 0x01

xcvr_reconfig_waitrequest

4.14. Page Request Service (PRS) Interface (EP Only)

When an Endpoint determines that it requires access to a page for which the ATS
translation is not available, it sends a Page Request message to request that the page
be mapped into system memory.

The PRS interface allows the monitoring of when PRS events happen, what functions
these PRS events belong to, and what types of events they are.

The PRS interface is only available in EP mode, and with TLP Bypass disabled.

Note: The P-Tile Avalon-ST IP for PCIe only provides the PRS capability. To take advantage of
this feature, you need to implement the necessary logic in your application.

Note: In the Intel Quartus Prime 20.3 release, only PF0 supports PRS. Furthermore, in this
release, the PRS interface only has Compilation (C) and Simulation (S) support.

Table 71. PRS Interface Signals

Signal Name Direction Description Clock Domain EP/RP/BP

prs_event_valid_i I This signal qualifies coreclkout_hip EP

prs_event_func_i
and prs_event_i.
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

102
4. Interfaces
UG-20225 | 2021.02.18

Signal Name Direction Description Clock Domain EP/RP/BP

There is a single-cycle
pulse for each PRS
event.

prs_event_func_i[ I The function number coreclkout_hip EP

2:0] for the PRS event.

prs_event_i[1:0] I 00 : Indicate that the coreclkout_hip EP

function has received
a PRG response
failure.
01: Indicate that the
function has received
a response with
Unexpected Page
Request Group Index.
10: Indicate that the
function has
completed all
previously issued page
requests and that it
has stopped requests
for additional pages.
Only valid when the
PRS enable bit is clear.
11: reserved.

The figure below shows the timing diagram for the PRS event interface when the
application layer of function 0 sends an event of PRG response reception, and the
application layer of function 1 sends an event stopping requests for additional pages.

Figure 53. Example Timing Diagram for the PRS Event Interface

coreclkout_hip

prs_event_valid_i

prs_event_func_i[2:0] 0x0 0x1

prs_event_i[1:0] 0x0 0x2

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

103
 UG-20225 | 2021.02.18

Send Feedback

5. Advanced Features

5.1. PCIe Port Bifurcation and PHY Channel Mapping

The PCIe controller IP contains a set of port bifurcation muxes to remap the four
controller PIPE lane interfaces to the shared 16 PCIe PHY lanes. The table below shows
the relationship between PHY lanes and the port mapping.

Table 72. Port Bifurcation and PHY Channel Mapping

Bifurcation Mode Port 0 (x16) Port 1 (x8) Port 2 (x4) Port 3 (x4)

1 x16 0 - 15 NA NA NA

2 x8 0-7 8 - 15 NA NA

4 x4 4-7 8 - 11 0-3 12 - 15

Note: For more details on the bifurcation modes, refer to the Architecture section in chapter
2.

5.2. Virtualization Support

The two components of the P-Tile IP for PCIe's virtualization support are:
• Single root I/O virtualization (SR-IOV)
• VirtIO

5.2.1. SR-IOV Support

The P-Tile IP for PCIe supports SR-IOV. The endpoint port controllers in the IP support
up to eight physical functions (PF) and 2048 virtual functions (VF) per SR-IOV
endpoint. The VF configuration space registers are hardened in the P-Tile. The specific
VF-based work queues and interrupt tables must be implemented in the FPGA fabric
by the user application.

For more details on the configuration space registers required for virtualization
support, refer to Configuration Space Registers for Virtualization on page 192.

5.2.1.1. SR-IOV Supported Features List

Table 73. SR-IOV Supported Features

Feature Support

SR-IOV Supported in x16/x8 controller EP mode.

continued...

Intel Corporation. All rights reserved. Agilex, Altera, Arria, Cyclone, Enpirion, Intel, the Intel logo, MAX, Nios,
Quartus and Stratix words and logos are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or
other countries. Intel warrants performance of its FPGA and semiconductor products to current specifications in ISO
accordance with Intel's standard warranty, but reserves the right to make changes to any products and services 9001:2015
at any time without notice. Intel assumes no responsibility or liability arising out of the application or use of any Registered
information, product, or service described herein except as expressly agreed to in writing by Intel. Intel
customers are advised to obtain the latest version of device specifications before relying on any published
information and before placing orders for products or services.
*Other names and brands may be claimed as the property of others.
5. Advanced Features
UG-20225 | 2021.02.18

Feature Support

Not supported in RP mode (x4).

MSI Supported in PFs only. Not supported in VFs.

No Per Vector Masking (PVM). If you need PVM, you must
use MSI-X.
Note: When SR-IOV is enabled, either MSI or MSI-X must
be enabled.

MSI-X Supported by all PFs.

For SR-IOV, PFs and VFs are always MSI-X capable.
Note: VFs share a common Table Size. VF Table BIR/Offset
and PBA BIR/Offset are fixed at compile time.
Note: When SR-IOV is enabled, either MSI or MSI-X must
be enabled.

Function Level Reset (FLR) Supported by all PFs/VFs.

Required for all SR-IOV functions.

Extended Tags Supported by all PFs/VFs. The Extended Tags feature allows
the TLP Tag field to be 8-bit, thus allowing the support of
256 tags.
Note that the application is restricted to a max of 256
outstanding tags, at any given time, for all functions
combined.
The application logic is responsible for implementing the tag
generation/tracking functions.
This feature is reflected in the Extended Tag Field
Supported in the Device Capabilities register. By default,
this field is set to 1 in every physical function enabled in the
Intel FPGA P-Tile IP for PCI Express.

10-bit Tags Supported by all PFs/VFs. Refer to Tag Allocation on page 69

for more details.

AER PFs are always AER capable. No AER implemented for VFs.

Active-State Power Management (ASPM) Optionality Supported by all PFs/VFs.

Compliance Only used to indicate ASPM is not supported.

Atomic Ops Requester capability is supported by all PFs/VFs.

Completer capability is supported.
Compare and Swap (CAS) AtomicOps are also supported.
They can handle up to 128-bit operands.

Internal Error Reporting Supported by all PFs (because all PFs are AER capable). No
support for VFs (because VFs do not support AER).

TLP Processing Hints 2-bit Processing Hint and 8-bit Steering Tag are supported
by all PFs/VFs. TPH Prefixes are not supported.
You can optionally choose to enable the TPH Requestor
capability. However, the IP is always TPH Completer
capable.

ID-Based ordering Supported by all PFs/VFs.

However, the IP core does NOT perform the reordering. The
Application Layer must do this.
The IP core only provides the IDO Request & Completion
Enable bits in the Device Control 2 register. This gives the
application permission to set the Attr bits in Requests and
Completions that it transmits.
Note: Reordering capability on the RX side may be limited
by your bypass queue. On the TX side, the IP core
does not set the IDO bits on internally generated
TLPs.
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

105
 5. Advanced Features
UG-20225 | 2021.02.18

Feature Support

Relaxed Ordering Implemented on the RX side. This feature is always active.

On the TX side, reordering is done by the application.

Alternative Routing ID Interpretation (ARI) EP (PFs/VFs) is always ARI capable. This is a device-level
option (all lanes or none will support ARI).
In addition, RP will always be ARI capable (ARI Forwarding
Supported bit is always 1).

Address Translation Service (ATS) Supported by all EP PFs/VFs.

Page Request Service Interface (PRI) Supported by all EP PFs/VFs.

User Extensions (Customer VSEC) Supported by all PFs/VFs.

Gen3 Receiver Impedance (3.0 ECN) Supported

Device Serial Number Supported

Completion Timeout Ranges (Device Capabilities 2) All ranges are supported.

Data Link Layer Active Reporting Capability (Link This capability is always supported in RP mode, but not in
Capabilities) EP mode.

Surprise Down Error Reporting Capability (Link Capabilities) Supported

PM-PCI Power Management Only D0/D3 states are supported.

ASPM (L0s/L1) Not supported

Process Address Space ID (PASID) Supported

TLP prefix Supported, mainly for PASID

Latency Tolerance Reporting (LTR) Supported (only for PASID)

Access Control Services Supported

5.2.1.2. Implementation

The VF configuration space is implemented in P-Tile logic, and does not require FPGA
fabric resources.

Accessing VF PCIe Information:

Due to the limited number of pins between P-Tile and the FPGA fabric, the PCIe
configuration space for VFs is not directly available to the user application.

User application can use the following methods to retrieve necessary information (bus
master enable, MSI-X etc…):
• Monitor specific VF registers using the Configuration Intercept Interface (for more
details, refer to section Configuration Intercept Interface (EP Only) on page 91).
• Read/write specific VF registers using the Hard IP Reconfiguration Interface (for
more details, refer to Targeting VF Configuration Space Registers in section Using
Direct User Avalon-MM Interface (Byte Access) on page 98).

Accessing VF PCIe Information:

VF IDs are calculated within P-Tile. User application has sideband signals
rx_st_vf_num_o and rx_st_vf_active_o with the TLP to identify the associated
VFs within the PFs.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

106
5. Advanced Features
UG-20225 | 2021.02.18

BDF Assignments:

When SR-IOV is enabled, the ARI capability is always enabled.

The P-Tile IP for PCIe automatically calculates the completer/requester ID on the

Transmit side.

User application needs to provide the VF and PF information in the Header as shown
below:

(For X16, sn is either s0 or s1. For X8, sn is s0).

• tx_st_hdr_sn[127]: must be set to 0
• tx_st_hdr_sn[83]: tx_st_vf_active
• tx_st_hdr_sn[82:80]: tx_st_func_num[2:0]
• tx_st_hdr_sn[95:84]: tx_st_vf_num[11:0]

In the following example, VF3 of PF1 is receiving and sending a request:

For the Receive TLP:

rx_st_func_num_o = 1h indicating that a VF associated with PF1 is making the

request.

rx_st_vf_num_o = 3h, and rx_st_vf_active_o = 1 indicating that VF3 of PF1 is

the active VF.

For the Transmit TLP of VF3 associated with PF1:

• tx_st_hdr_sn[83] = 1h
• tx_st_hdr_sn[82:80] = 1h
• tx_st_hdr_sn[95:84] = 3h

5.2.1.2.1. VF Error Flag Interface (for x16/x8 Cores Only)

The VFs, with no AER support, are required to generate Non-Fatal error messages. The
IP does not generate any error message. It is up to the user application logic to
generate appropriate messages when specific error conditions occur.

The P-Tile IP for PCIe makes necessary signals available to the user application logic
to generate these messages. The Completion Timeout Interface (described in section
Completion Timeout Interface on page 81) and the signals listed in the table below
provide the necessary information to generate Non-Fatal error messages.

Table 74. VF Error Flag Interface

Signal Name Direction Description Clock Domain EP/RP/BP

X16: O Indicates a Poisoned Write Request is coreclkout_hip EP

vf_err_poisonedwr received.
req_s0/1/2/3_o
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

107
 5. Advanced Features
UG-20225 | 2021.02.18

Signal Name Direction Description Clock Domain EP/RP/BP

X8:
vf_err_poisonedwr
req_s0/1_o

X16: O Indicates a Poisoned Completion is coreclkout_hip EP

vf_err_poisonedco received.
mpl_s0/1/2/3_o
X8:
vf_err_poisonedco
mpl_s0/1_o

X16: O Indicates the IP core received a Posted coreclkout_hip EP

vf_err_ur_posted_ UR request.
s0/1/2/3_o
X8:
vf_err_ur_posted_
s0/1_o

X16: O Indicates the IP core received a Posted coreclkout_hip EP

vf_err_ca_postedr CA request.
eq_s0/1/2/3_o
X8:
vf_err_ca_postedr
eq_s0/1_o

X16: O Indicates the VF number for which the coreclkout_hip EP

vf_err_vf_num_s0/ error is detected.
1/2/3_o[10:0]
X8:
vf_err_vf_num_s0/
1_o[10:0]

X16: O Indicates the physical function number coreclkout_hip EP

vf_err_func_num_s associated with the VF that has the
0/1/2/3_o[2:0] error.
X8:
vf_err_func_num_s
0/1_o[2:0]

vf_err_overflow_o O Indicates a VF error FIFO overflow and coreclkout_hip EP

a loss of an error report. The overflow
can happen when coreclkout_hip is
slower than the default value. If
coreclkout_hip is running at the
default frequency, the overflow will not
happen.

user_sent_vfnonfa I Indicates the user application sent a coreclkout_hip EP

talmsg_s0_i non-fatal error message in response to
an error detected.

user_vfnonfatalms I Indicates the VF number for which the coreclkout_hip EP

g_vfnum_s0_i[10:0 error message was generated. This bus
] is valid when
user_sent_vfnonfatalmsg_s0_i is
high.

user_vfnonfatalms I Indicates the PF number associated coreclkout_hip EP

g_func_num_s0_i[2 with the VF with the error. This bus is
:0] valid when
user_sent_vfnonfatalmsg_s0_i is
high.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

108
5. Advanced Features
UG-20225 | 2021.02.18

5.2.1.3. VF to PF Mapping

VF to PF mapping always starts from the lowest possible PF number. For instance, if
the IP has 2 PFs, wherein PF0 has 64 VFs and PF1 has 16 VFs, VF1 to VF64 are
mapped to PF0, and VF65 to VF80 are mapped to PF1.

Currently, the IP core only supports the following PF/VF combinations:

Table 75. Supported PF/VF Combinations

Number of VFs per PF
Number of PFs Total VFs
(PF0/PF1/PF2/PF3/PF4/PF5/PF6/PF7)

1 8 8

1 16 16

1 32 32

1 64 64

1 128 128

1 256 256

1 512 512

2 16/16 32

2 32/32 64

2 128/128 256

2 256/256 512

2 32/0 32

2 0/32 32

2 64/0 64

2 0/64 64

2 128/0 128

2 0/128 128

2 256/0 256

2 0/256 256

2 512/0 512

2 0/512 512

2 1024/0 1024

2 0/1024 1024

2 2048/0 2048

2 0/2048 2048

4 128/0/0/0 128

4 0/128/0/0 128

4 256/0/0/0 256

4 0/256/0/0 256
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

109
 5. Advanced Features
UG-20225 | 2021.02.18

Number of VFs per PF

Number of PFs Total VFs
(PF0/PF1/PF2/PF3/PF4/PF5/PF6/PF7)

4 1024/0/0/0/0 1024

4 0/1024/0/0 1024

8 256/0/0/0/0/0/0/0 256

8 0/256/0/0/0/0/0/0 256

For example, the row that shows the combination of four PFs, 256 VFs, and the
notation 256/0/0/0 in the Number of VFs per PF column indicates that all 256
VFs are mapped to PF0, while no VF is mapped to PF1, PF2 or PF3.

Note: SR-IOV permutations allow any PF to be assigned the initial VF allocation.

5.2.1.4. Function Level Reset (FLR)

Use the FLR interface to reset individual SR-IOV functions. The PCIe Hard IP supports
FLR for both PFs and VFs. If the FLR is for a specific VF, the received packets for that
VF are no longer valid. The flr_* interface signals are provided to the application
interface for this purpose. When the flr_rcvd* signal is asserted, it indicates that an
FLR is received for a particular PF/VF. Application logic needs to perform its FLR
routine and send the completion status back on the flr_completed* interface. The
Hard IP will wait for the flr_completed* status to re-enable the VF. Prior to that event,
the Hard IP will respond to all transactions to the function that is reset by the FLR with
completions with an Unsupported Request (UR) status.

The following figure shows the timing diagram for an FLR event targeting a PF (PF2 in
this example):

Figure 54. FLR for PF

p0_flr_rcvd_pf_o[7:0] 00 01
p0_flr_completed_pf_i[7:0] 00 01
coreclkout_hip

Here is the timing diagram for an FLR event targeting a VF:

Figure 55. FLR for VF

p1_flr_rcvd_vf_o
p1_flr_rcvd_pf_num_o[2:0] 0 2
p1_flr_rcvd_vf_num_o[10:0] 000 006
p1_flr_completed_vf_i
p1_flr_completed_vf_num_i[10:0] 000 006
p1_flr_completed_pf_num_i[2:0] 0 2

5.2.2. VirtIO Support

Note: In the 20.3 release of Intel Quartus Prime, only compilation and simulation are
supported for the VirtIO feature.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

110
5. Advanced Features
UG-20225 | 2021.02.18

5.2.2.1. VirtIO Supported Features List

• VirtIO devices are implemented as PCI Express devices.

• Support 8 PFs and 2K VFs VirtIO capability structure for each EP.
• Configuration Intercept Interface in the P-Tile IP for PCIe (EP mode only) is
provided for VirtIO transport.
• Five VirtIO device configuration structures are supported:
— Common configuration
— Notifications
— ISR Status
— Device-specific configuration (optional)
— PCI configuration access
• Location of each structure is specified using a vendor-specific PCI capability
located in the PCI configuration space of the device.
• VirtIO capability structure uses little-endian format.
• All fields of the VirtIO capability structure are read-only for the driver by default.
• Support PFs and VFs FLR
• Supports x16 and x8 cores.
• MSI is not supported with VirtIO.

5.2.2.2. Overview

The VirtIO PCI configuration access capability creates an alternative access method to
the common configuration, notifications, ISR, and device-specific configuration
structure regions. This interface provides a means for the driver to access the VirtIO
device region of Physical Functions (PFs) or Virtual Functions (VFs).

VirtIO is an industry standard for software-based virtualization that is supported

natively by Linux. In VirtIO, software implements the virtualization stack, whereas in
the case of SR-IOV, this stack is implemented mostly in hardware.

Below is the block diagram of the Soft IP which implements the VirtIO capability for
PFs and VFs. This Soft IP block is automatically included when the VirtIO feature is
enabled in the IP Parameter Editor.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

111
 5. Advanced Features
UG-20225 | 2021.02.18

Figure 56. VirtIO Soft IP Block Diagram

flr_rcvd*
coreclkout_hip
cfg_monitor User
p0_reset_status_n
Side

Addr DC Write Path for RW pci_cfg_access regs

cii_req (cs for reg
cii_func_num bank select) VirtIO
cii_dout pf_sel
PCI
cii_wr_vf vf_sel Read path for all Virtio regs Config
cii_vf_num Access
cii_addr
PF1 VirtIO Cap Reg Interface
Rd Wr Ctrl PF2_VFs
PF2
VirtIO
Cap Reg
PF8

...
cii_halt

.
cii_override_en cfg_rd_data
cii_override_din cfg_req_ack
Dataflow w/
Control ARBT PF8_VFs

...
(CII_o SM)

5.2.2.3. Parameters

For a detailed discussion of the VirtIO-related parameters, refer to the section VirtIO
Parameters on page 30 in the Parameters chapter.

5.2.2.4. VirtIO PCI Configuration Access Interface

To access a VirtIO device region, pci_cfg_data will provide a window of size cap.length
(1, 2 or 4 Bytes) into the given cap.bar (0x0 – 0x5) at offset cap.offset (multiple of
cap.length). Detailed interfaces mapping for the user application logic are shown in
the following table.

As for the VirtIO device, upon detecting a driver write access to pci_cfg_data, the user
application side's VirtIO device must execute a write access at cap.offset at the BAR
selected by cap.bar using the first cap.length bytes from pci_cfg_data. Moreover, upon
detecting a driver read access to pci_cfg_data, the user application side's VirtIO
device must execute a read access of length cap.length at cap.offset at the BAR
selected by cap.bar and store the first cap.length bytes in pci_cfg_data.

Table 76. VirtIO PCI Configuration Access Interface

Name Direction Description Clock Domain

virtio_pcicfg_vfaccess O Indicates the driver access is to a VF. coreclkout_hip

_o
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

112
5. Advanced Features
UG-20225 | 2021.02.18

Name Direction Description Clock Domain

The corresponding Virtual Function is

identified from the value of
virtio_pcicfg_vfnum_o.

virtio_pcicfg_vfnum_o[ O Indicates the corresponding Virtual Function coreclkout_hip

VFNUM_WIDTH-1:0] number associated with the current Physical
Function that the driver’s write or read
access is targeting.
Validated by virtio_pcicfg_vfaccess_o
and by driver write access to pci_cfg_data,
or driver read access to pci_cfg_data.

virtio_pcicfg_pfnum_o[ O Indicates the corresponding Physical coreclkout_hip

PFNUM_WIDTH-1:0] Function number that the driver’s write or
read access is targeting.
Validated by driver write access to
pci_cfg_data, or driver read access to
pci_cfg_data.

virtio_pcicfg_bar_o[7: O Indicates the BAR holding the PCI coreclkout_hip

0] configuration access structure. The driver
sets the BAR to access by writing to cap.bar.
Values 0x0 to 0x5 specify a BAR belonging
to the function located beginning at 10h in
the PCI Configuration Space. The BAR can
be either 32-bit or 64-bit.
Validated by driver write access to
pci_cfg_data, or driver read access to
pci_cfg_data.
The corresponding PF or VF is identified
from the value of virtio_pcicfg_p/
vfnum_o.

virtio_pcicfg_length_ O Indicates the length of the structure. The coreclkout_hip

o[31:0] length may include padding, or fields
unused by the driver, or future extensions.
The driver sets the size of the access by
writing 1, 2 or 4 to cap.length.
Validated by driver write access to
pci_cfg_data, or driver read access to
pci_cfg_data.
The corresponding PF or VF is identified
from the value of virtio_pcicfg_p/
vfnum_o.

virtio_pcicfg_baroffse O Indicates where the structure begins relative coreclkout_hip

t_o[31:0] to the base address associated with the
BAR. The driver sets the offset within the
BAR by writing to cap.offset.
Validated by driver write access to
pci_cfg_data, or driver read access to
pci_cfg_data.
The corresponding PF or VF is identified
from the value of virtio_pcicfg_p/
vfnum_o.

virtio_pcicfg_cfgdata_ O Indicates the data for BAR access. The coreclkout_hip

o[31:0] pci_cfg_data will provide a window of size
cap.length into the given cap.bar at offset
cap.offset.
Validated by driver write access to
pci_cfg_data, or driver read access to
pci_cfg_data.
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

113
 5. Advanced Features
UG-20225 | 2021.02.18

Name Direction Description Clock Domain

The corresponding PF or VF is identified

from the value of virtio_pcicfg_p/
vfnum_o.

virtio_pcicfg_cfgwr_o O Indicates driver write access to coreclkout_hip

pci_cfg_data.
The corresponding PF or VF is identified
from the value of virtio_pcicfg_p/
vfnum_o.

virtio_pcicfg_cfgrd_o O Indicates driver read access to pci_cfg_data. coreclkout_hip

The corresponding PF or VF is identified
from the value of virtio_pcicfg_p/
vfnum_o.

virtio_pcicfg_appvfnum I Indicates the corresponding Virtual Function coreclkout_hip

_i[VFNUM_WIDTH-1:0] number associated with the current Physical
Function that the application config data
storage is for.
Validated by virtio_pcicfg_rdack_i.

virtio_pcicfg_apppfnum I Indicates the corresponding Physical coreclkout_hip

_i[PFNUM_WIDTH-1:0] Function number that the application config
data storage is for.
Validated by virtio_pcicfg_rdack_i.

virtio_pcicfg_rdack_i I Indicates an application read access data coreclkout_hip

ack to store the config data in pci_cfg_data.
Usually the reasonable ack latency is no
more than 10 cycles.
The corresponding Virtual Function is
identified from the value of
virtio_pcicfg_appvfnum_i.

virtio_pcicfg_rdbe_i[3 I Indicates application enabled bytes within coreclkout_hip

:0] virtio_pcicfg_data_i.
Validated by virtio_pcicfg_rdack_i.
The corresponding Virtual Function is
identified from the value of
virtio_pcicfg_appvfnum_i.

virtio_pcicfg_data_i[3 I Indicates application data to be stored in coreclkout_hip

1:0] PCI Configuration Access data registers.
Validated by virtio_pcicfg_rdack_i
and virtio_pcicfg_rdbe_i.
The corresponding Virtual Function is
identified from the value of
virtio_pcicfg_appvfnum_i.

5.2.2.5. Registers

The following VirtIO capability structure registers references apply to each PF and VF.
Addresses shown are register addresses.

Table 77. PF/VF Capability Link List

Capability Start Byte Address Last Byte Address DW Count

Type0 0x00 0x3F 16

PM (PF only) 0x40 0x47 2

continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

114
5. Advanced Features
UG-20225 | 2021.02.18

Capability Start Byte Address Last Byte Address DW Count

VirtIO Common 0x48 0x57 4

Configuration

VirtIO Notifications 0x58 0x6B 5

Reserved 0x6C 0x6F 1

PCIe 0x70 0xA3 13

Reserved 0xA4 0xAF 3

MSIX 0xB0 0xBB 3

VirtIO ISR Status 0xBC 0xCB 4

VirtIO Device-Specific 0xCC 0xDB 4

Configuration

VirtIO PCI Configuration 0xDC 0xEF 5

Access

Reserved 0xF0 0xFF 4

Table 78. VirtIO Common Configuration Capability Structure

Address Name Description

012 Common Configuration Capability Capability ID, next capability pointer, capability length
Register

013 BAR Indicator Register Lower 8 bits indicate which BAR holds the structure

014 BAR Offset Register Indicates starting address of the structure within the BAR

015 Structure Length Register Indicates length of structure

VirtIO Notifications Capability Structure

016 Notifications Capability Register Capability ID, next capability pointer, capability length

017 BAR Indicator Register Lower 8 bits indicate which BAR holds the structure

018 BAR Offset Register Indicates starting address of the structure within the BAR

019 Structure Length Register Indicates length of structure

01A Notify Off Multiplier Multiplier for queue_notify_off

VirtIO ISR Status Capability Structure

02F ISR Status Capability Register Capability ID, next capability pointer, capability length

030 BAR Indicator Register Lower 8 bits indicate which BAR holds the structure

031 BAR Offset Register Indicates starting address of the structure within the BAR

032 Structure Length Register Indicates length of structure

VirtIO Device-Specific Capability Structure (Optional)

033 Device Specific Capability Register Capability ID, next capability pointer, capability length

034 BAR Indicator Register Lower 8 bits indicate which BAR holds the structure

035 BAR Offset Register Indicates starting address of the structure within the BAR

036 Structure Length Register Indicates length of structure

VirtIO PCI Configuration Access Structure

continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

115
 5. Advanced Features
UG-20225 | 2021.02.18

Address Name Description

037 PCI Configuration Access Capability Capability ID, next capability pointer, capability length
Register

038 BAR Indicator Register Lower 8 bits indicate which BAR holds the structure

039 BAR Offset Register Indicates starting address of the structure within the BAR

03A Structure Length Register Indicates length of structure

03B PCI Configuration Data Data for BAR access

5.2.2.5.1. VirtIO Common Configuration Capability Register (Address: 0x012)

The capability register identifies that this is a vendor-specific capability. It also

identifies the structure type.

Table 79. VirtIO Common Configuration Capability Register

Bit Location Description Access Type Default Value

31:24 Configuration Type RO 0x01

23:16 Capability Length RO 0x10

15:8 Next Capability Pointer RO 0x58

7:0 Capability ID RO 0x09

5.2.2.5.2. VirtIO Common Configuration BAR Indicator Register (Address: 0x013)

The Bar Indicator field holds the values 0x0 to 0x5 specifying a Base Address register
(BAR) belonging to the function located beginning at 10h in PCI Configuration Space.
The BAR is used to map the structure into the memory space. Any other value is
reserved for future use.

Table 80. VirtIO Common Configuration BAR Indicator Register

Bit Location Description Access Type Default Value

31:24 Padding RO 0x00

23:16 Padding RO 0x00

15:8 Padding RO 0x00

7:0 BAR Indicator RO Settable through Platform

Designer

5.2.2.5.3. VirtIO Common Configuration BAR Offset Register (Address: 0x014)

This register indicates where the structure begins relative to the base address
associated with the BAR. The alignment requirements of the offset are indicated in
each structure-specific section.

Table 81. VirtIO Common Configuration BAR Offset Register

Bit Location Description Access Type Default Value

31:0 BAR Offset RO Settable through Platform

Designer

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

116
5. Advanced Features
UG-20225 | 2021.02.18

5.2.2.5.4. VirtIO Common Configuration Structure Length Register (Address 0x015)

The length register indicates the length of the structure. The length may include
padding, fields unused by the driver, or future extensions.

Table 82. VirtIO Common Configuration Structure Length Register

Bit Location Description Access Type Default Value

31:0 Structure Length RO Settable through Platform

Designer

5.2.2.5.5. VirtIO Notifications Capability Register (Address: 0x016)

The capability register identifies that this is a vendor-specific capability. It also

identifies the structure type.

Table 83. VirtIO Notifications Capability Register

Bit Location Description Access Type Default Value

31:24 Configuration Type RO 0x02

23:16 Capability Length RO 0x14

15:8 Next Capability Pointer RO 0xBC

7:0 Capability ID RO 0x09

5.2.2.5.6. VirtIO Notifications BAR Indicator Register (Address: 0x017)

The Bar Indicator field holds the values 0x0 to 0x5 specifying a Base Address register
(BAR) belonging to the function located beginning at 10h in PCI Configuration Space.
The BAR is used to map the structure into memory space. Any other value is reserved
for future use.

Table 84. VirtIO Notifications BAR Indicator Register

Bit Location Description Access Type Default Value

31:24 Padding RO 0x00

23:16 Padding RO 0x00

15:8 Padding RO 0x00

7:0 BAR Indicator RO Settable through Platform

Designer

5.2.2.5.7. VirtIO Notifications BAR Offset Register (Address: 0x018)

This register indicates where the structure begins relative to the base address
associated with the BAR. The alignment requirements of the offset are indicated in
each structure-specific section.

Table 85. VirtIO Notifications BAR Offset Register

Bit Location Description Access Type Default Value

31:0 BAR Offset RO Settable through Platform

Designer

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

117
 5. Advanced Features
UG-20225 | 2021.02.18

5.2.2.5.8. VirtIO Notifications Structure Length Register (Address: 0x019)

The length register indicates the length of the structure. The length may include
padding, fields unused by the driver, or future extensions.

Table 86. VirtIO Notifications Structure Length Register

Bit Location Description Access Type Default Value

31:0 Structure Length RO Settable through Platform

Designer

5.2.2.5.9. VirtIO Notifications Notify Off Multiplier Register (Address: 0x01A)

The notify off multiplier register indicates the multiplier for queue_notify_off in the
structure.

Table 87. VirtIO Notifications Notify Off Multiplier Register

Bit Location Description Access Type Default Value

31:0 Multiplier for RO Settable through Platform

queue_notify_off Designer

5.2.2.5.10. VirtIO ISR Status Capability Register (Address: 0x02F)

The capability register identifies that this is a vendor-specific capability. It also

identifies the structure type.

Table 88. VirtIO ISR Status Capability Register

Bit Location Description Access Type Default Value

31:24 Configuration Type RO 0x03

23:16 Capability Length RO 0x10

15:8 Next Capability Pointer RO If Device-Specific Capability

is present, then points to
0xCC, else points to 0xDC.

7:0 Capability ID RO 0x09

5.2.2.5.11. VirtIO ISR Status BAR Indicator Register (Address: 0x030)

The Bar Indicator field holds the values 0x0 to 0x5 specifying a Base Address register
(BAR) belonging to the function located beginning at 10h in PCI Configuration Space.
The BAR is used to map the structure into memory space. Any other value is reserved
for future use.

Table 89. VirtIO ISR Status BAR Indicator Register

Bit Location Description Access Type Default Value

31:24 Padding RO 0x00

23:16 Padding RO 0x00

15:8 Padding RO 0x00

7:0 BAR Indicator RO Settable through Platform

Designer

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

118
5. Advanced Features
UG-20225 | 2021.02.18

5.2.2.5.12. VirtIO ISR Status BAR Offset Register (Address: 0x031)

This register indicates where the structure begins relative to the base address
associated with the BAR. The alignment requirements of the offset are indicated in
each structure-specific section.

Table 90. VirtIO ISR Status BAR Offset Register

Bit Location Description Access Type Default Value

31:0 BAR Offset RO Settable through Platform

Designer

5.2.2.5.13. VirtIO ISR Status Structure Length Register (Address: 0x032)

The length register indicates the length of the structure. The length may include
padding, fields unused by the driver, or future extensions.

Table 91. VirtIO ISR Status Structure Length Register

Bit Location Description Access Type Default Value

31:0 Structure Length RO Settable through Platform

Designer

5.2.2.5.14. VirtIO Device Specific Capability Register (Address: 0x033)

The capability register identifies that this is vendor-specific capability. It also identifies
the structure type.

Table 92. VirtIO Device Specific Capability Register

Bit Location Description Access Type Default Value

31:24 Configuration Type RO 0x04

23:16 Capability Length RO 0x10

15:8 Next Capability Pointer RO If this capability is present,

then points to 0xDC.

7:0 Capability ID RO 0x09

5.2.2.5.15. VirtIO Device Specific BAR Indicator Register (Address: 0x034)

The BAR Indicator field holds the values 0x0 to 0x5 specifying a Base Address register
(BAR) belonging to the function located beginning at 10h in PCI Configuration Space.
The BAR is used to map the structure into memory space. Any other value is reserved
for future use.

Table 93. VirtIO Device Specific BAR Indicator Register

Bit Location Description Access Type Default Value

31:24 Padding RO 0x00

23:16 Padding RO 0x00

15:8 Padding RO 0x00

7:0 BAR Indicator RO Settable through Platform

Designer

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

119
 5. Advanced Features
UG-20225 | 2021.02.18

5.2.2.5.16. VirtIO Device Specific BAR Offset Register (Address 0x035 )

This register indicates where the structure begins relative to the base address
associated with the BAR. The alignment requirements of the offset are indicated in
each structure-specific section.

Table 94. VirtIO Device Specific BAR Offset Register

Bit Location Description Access Type Default Value

31:0 BAR Offset RO Settable through Platform

Designer

5.2.2.5.17. VirtIO Device Specific Structure Length Register (Address: 0x036)

The length register indicates the length of the structure. The length may include
padding, fields unused by the driver, or future extensions.

Table 95. VirtIO Device Specific Structure Length Register

Bit Location Description Access Type Default Value

31:0 Structure Length RO Settable through Platform

Designer

5.2.2.5.18. VirtIO PCI Configuration Access Capability Register (Address: 0x037)

The capability register identifies that this is a vendor-specific capability. It also

identifies the structure type.

Table 96. VirtIO PCI Configuration Access Capability Register

Bit Location Description Access Type Default Value

31:24 Configuration Type RO 0x05

23:16 Capability Length RO 0x14

15:8 Next Capability Pointer RO 0x00

7:0 Capability ID RO 0x09

5.2.2.5.19. VirtIO PCI Configuration Access BAR Indicator Register (Address: 0x038)

The BAR Indicator field holds the values 0x0 to 0x5 specifying a Base Address register
(BAR) belonging to the function located beginning at 10h in PCI Configuration Space.
The BAR is used to map the structure into memory space. Any other value is reserved
for future use.

Table 97. VirtIO PCI Configuration Access BAR Indicator Register

Bit Location Description Access Type Default Value

31:24 Padding RO 0x00

23:16 Padding RO 0x00

15:8 Padding RO 0x00

7:0 BAR Indicator RW Settable through Platform

Designer

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

120
5. Advanced Features
UG-20225 | 2021.02.18

5.2.2.5.20. VirtIO PCI Configuration Access BAR Offset Register (Address: 0x039)

This register indicates where the structure begins relative to the base address
associated with the BAR. The alignment requirements of the offset are indicated in
each structure-specific section.

Table 98. VirtIO PCI Configuration Access BAR Offset Register

Bit Location Description Access Type Default Value

31:0 BAR Offset RW Settable through Platform

Designer

5.2.2.5.21. VirtIO PCI Configuration Access Structure Length Register (Address: 0x03A)

The length register indicates the length of the structure. The length may include
padding, fields unused by the driver, or future extensions.

Table 99. VirtIO PCI Configuration Access Structure Length Register

Bit Location Description Access Type Default Value

31:0 Structure Length RW Settable through Platform

Designer

5.2.2.5.22. VirtIO PCI Configuration Access Data Register (Address: 0x03B)

The PCI configuration data register indicates the data for BAR access.

Table 100. VirtIO PCI Configuration Access Data Register

Bit Location Description Access Type Default Value

31:0 PCI Configuration Data RW Settable through Platform

Designer

5.3. TLP Bypass Mode

The P-Tile Avalon-ST IP for PCIe includes a TLP Bypass mode for both downstream and
upstream ports to allow the implementation of advanced features such as:
• The upstream port or the downstream port of a switch.
• A custom implementation of a Transaction Layer to meet specific user
requirements.

Table 101. Supported TLP Bypass Configurations

UP = upstream port; DN = downstream port

IP Mode Port Mode

X16 UP
DN

X8 UP/UP
UP/DN
DN/UP
DN/DN

X4 UP/UP/UP/UP
DN/DN/DN/DN

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

121
 5. Advanced Features
UG-20225 | 2021.02.18

5.3.1. Overview
When the TLP Bypass feature is enabled, the P-Tile Avalon-ST IP does not process
received TLPs internally but outputs them to the user application. This allows the
application to implement a custom Transaction Layer.

Note that in TLP Bypass mode, the PCIe Hard IP does not generate/check the ECRC
and will not remove it if the received TLP has the ECRC.

The P-tile Avalon-ST IP in TLP Bypass mode still includes some of the PCIe
configuration space registers related to link operation (refer to the Configuration
Space Registers on page 184 chapter for the list of registers).

P-Tile interfaces with the application logic via the Avalon-ST interface (for all TLP
traffic), the User Avalon-MM interface (for Lite TL’s configuration registers access) and
other miscellaneous signals.

Figure 57. P-Tile Avalon-ST IP in TLP Bypass Mode

Hard IP in TLP BYPASS

Rx
Buffer Lite Transaction Layer

Avalon ST

PMA + PCS PHY Layer Data Link Layer Lite PCle

MISC
Configuration

User Avalon MM
Replay
Buffer

In TLP bypass mode, P-Tile supports the autonomous Hard IP feature. It responds to
configuration accesses before the FPGA fabric enters user mode with Completions with
a CRS code.

However, in TLP bypass mode, CvP init and update are not supported.

5.3.2. Register Settings for the TLP Bypass Mode

When TLP Bypass mode is enabled, some error detection is still performed in the
Physical and Link Layers inside the Hard IP. Per PCIe specification, the Hard IP must
report these errors on the configuration space registers (in the AER Capability
Structure). The P-tile IP for PCIe includes two registers called TLPBYPASS_ERR_EN
and TLPBYPASS_ERR_STATUS to report errors detected while in TLP Bypass mode.

TLPBYPASS_ERR_EN and TLPBYPASS_ERR_STATUS are part of the configuration and

status register.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

122
5. Advanced Features
UG-20225 | 2021.02.18

5.3.2.1. TLPBYPASS_ERR_EN (Address 0x104194)

This register allows you to enable or disable error reporting. When this feature is
disabled, the TLPBYPASS_ERR_ STATUS bits associated with an error are not set when
the error is detected.

Table 102. TLPBYPASS_ERR_EN Register (Address 0x104194)

Name Bits Reset Value Access Mode Register Description

Reserved [31:20] 12’b0 RO Reserved

k_cfg_uncor_internal_e Enable error indication on serr_out_o

[19] 1'b1 RW
rr_sts_en for Uncorrectable Internal Error.

k_cfg_corrected_intern Enable error indication on serr_out_o

[18] 1'b1 RW
al_err_sts_en for Corrected Internal Error.

k_cfg_rcvr_overflow_er Enable error indication on serr_out_o

[17] 1'b1 RW
r_sts_en for Receiver Overflow Error.

k_cfg_fc_protocol_err_ Enable error indication on serr_out_o

[16] 1'b1 RW
sts_en for Flow Control Protocol Error.

k_cfg_mlf_tlp_err_sts_ Enable error indication on serr_out_o

[15] 1'b1 RW
en for Malformed TLP Error.

k_cfg_surprise_down_er Enable error indication on serr_out_o

[14] 1'b1 RW
r_sts_en for Surprise Down Error.

k_cfg_dl_protocol_err_ Enable error indication on serr_out_o

[13] 1'b1 RW
sts_en for Data Link Protocol Error.

k_cfg_replay_number_ro Enable error indication on serr_out_o

[12] 1'b1 RW
llover_err_sts_en for REPLAY_NUM Rollover Error.

k_cfg_replay_timer_tim Enable error indication on serr_out_o

[11] 1'b1 RW
eout_err_st_en for Replay Timer Timeout Error.

k_cfg_bad_dllp_err_sts Enable error indication on serr_out_o

[10] 1'b1 RW
_en for Bad DLLP Error.

k_cfg_bad_tlp_err_sts_ Enable error indication on serr_out_o

[9] 1'b1 RW
en for Bad TLP Error.

k_cfg_rcvr_err_sts_en Enable error indication on serr_out_o

[8] 1'b1 RW
for Receiver Error.

Reserved [7:1] 7'b0 RO Reserved

k_cfg_ecrc_err_sts_en Enable error indication on serr_out_o

[0] 1'b1 RW
for ECRC Error.

5.3.2.2. TLPBYPASS_ERR_STATUS (Address 0x104190)

When an error is detected, Intel recommends that you read the PF0 AER register
inside P-Tile to get detailed information about the error. To clear the previous error
status, you need to clear TLPBYPASS_ERR_STATUS and the corresponding correctable
and uncorrectable error status registers in the AER capability structure. After doing
that, you can get the new error update from this register.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

123
 5. Advanced Features
UG-20225 | 2021.02.18

Table 103. TLPBYPASS_ERR_STATUS Register (Address 0x104190)

Name Bits Reset Value Access Mode Register Description

Reserved [31:20] 12’b0 RO Reserved

cfg_uncor_internal_e Uncorrectable Internal Error

[19] 1'b0 W1C
rr_sts

cfg_corrected_intern Corrected Internal Error

[18] 1'b0 W1C
al_err_sts

cfg_rcvr_overflow_er Receiver Overflow Error

[17] 1'b0 W1C
r_sts

cfg_fc_protocol_err_ Flow Control Protocol Error

[16] 1'b0 W1C
sts

cfg_mlf_tlp_err_sts [15] 1'b0 W1C Malformed TLP Error

cfg_surprise_down_er Surprise Down Error. Available in

[14] 1'b0 W1C downstream mode only.
r_sts

cfg_dl_protocol_err_ Data Link Protocol Error

[13] 1'b0 W1C
sts

cfg_replay_number_ro REPLAY_NUM Rollover Error

[12] 1'b0 W1C
llover_err_sts

cfg_replay_timer_tim Replay Timer Timeout Error

[11] 1'b0 W1C
eout_err_sts

cfg_bad_dllp_err_sts [10] 1'b0 W1C Bad DLLP Error

cfg_bad_tlp_err_sts [9] 1'b0 W1C Bad TLP Error

cfg_rcvr_err_sts [8] 1'b0 W1C Receiver Error

Reserved [7:1] 7'b0 RO Reserved

cfg_ecrc_err_sts [0] 1'b0 W1C ECRC Error

5.3.3. User Avalon-MM Interface

For more details on the signals in this interface, refer to the section Hard IP
Reconfiguration Interface on page 93.

The majority of the PCIe standard registers are implemented in the user’s logic
outside of the P-Tile Avalon-ST IP.

However, the following registers still remain inside the P-Tile:

• Power management capability
• PCI Express capability
• Secondary PCI Express capability
• Data link feature extended capability
• Physical layer 16.0GT/s extended capability
• Lane margining at the receiver extended capability
• Advanced error reporting capability

The application can only access PCIe controller registers through the User Avalon-MM
interface.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

124
5. Advanced Features
UG-20225 | 2021.02.18

Table 104. Capability Registers to be Updated by the Application Logic via the User
Avalon-MM Interface
Capability Comments

Power Management Capability Need to write back since it is required to trigger a PCI-PM entry.

PCI Express Capability All the PCIe capabilities, control and status registers are for configuring
the device. Write-back is required.

Secondary PCI Express Capability Secondary PCIe Capability is required for configuring the device.

Data Link Feature Extended Capability Data Link Capability is device specific.

Physical Layer 16.0 GT/s Extended Capability Physical Layer 16G Capability is device specific.

Lane Margining at the Receiver Extended Margining Extended Capability is device specific.
Capability

Advanced Error Reporting Capability Write-back to error status registers is required for TLP Bypass.

Note: Refer to Table 119 on page 184 for the address offsets information for the capability
registers listed in the table above.

5.3.4. Avalon-ST Interface

For more details on the signals in this interface, refer to the section Avalon-ST
Interface on page 51.

5.3.4.1. Configuration TLP

The P-Tile IP forwards any received Type0/1 Configuration TLP to the Avalon-ST RX
streaming interface. User’s logic has the responsibility to respond with a Completion
TLP with a Completion code of Successful Completion (SC), Unsupported Request
(UR), Configuration Request Retry Status (CRS), or Completer Abort (CA).

If a Configuration TLP needs to update a register in the PCIe configuration space in the
P-Tile PCIe Hard IP, you need to use the User Avalon-MM interface.

The application needs to prevent link programming side effects such as writing into
low-power states before sending the Completion associated with the request. The
application logic can check the TX FIFO empty flag in the tx_cdts_limit_o after the
Completion enters the TX streaming interface to confirm that the TLP has been sent.
For more details on the User Avalon-MM interface, refer to the section Hard IP
Reconfiguration Interface (User Avalon-MM Interface).

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

125
 5. Advanced Features
UG-20225 | 2021.02.18

Figure 58. Configuration TLP Received by P-Tile IP for PCIe Targeting the Hard IP
Internal Registers
Hard IP in TLP BYPASS

Rx
Buffer Lite Transaction Layer RX Avalon ST
CFG Type 0
Targeting HIP
Registers
PMA + PCS PHY Layer Data Link Layer Lite PCle
Avalon MM
Configuration

Replay TX Avalon ST
Buffer
CPL Generated
After Avalon
MM Access

5.3.4.2. Transmit Interface

All TLPs transmitted by the application through the TX streaming interface are sent out
as-is, without any tracking for completion. The P-Tile IP for PCIe does not perform any
check on the TLPs. Your application logic is responsible for sending TLPs that comply
with the PCIe specifications.

5.3.4.3. Receive Interface

ALL TLPs received by the IP are transmitted to the application through the RX
streaming interface (except Malformed TLPs).

Please refer to the Packets Forwarded to the User Application in TLP Bypass Mode on
page 205 Appendix for detailed information.

All PCIe protocol errors leading up to designating a TLP packet as a good packet or not
will be detected by the Hard IP and communicated to user logic to take appropriate
action in terms of error logging and escalation. The IP does not generate any error
message internally, since this is the responsibility of the user logic.

5.3.4.4. Malformed TLP

In TLP Bypass mode, a malformed TLP is dropped in the P-Tile IP for PCIe and its
event is logged in the AER capability registers. P-Tile also notifies you of this event by
asserting the serr_out_o signal.

Refer to the PCI Express Base Specification for the definition of a malformed TLP.

5.3.4.5. ECRC
In TLP bypass mode, the ECRC is not generated or stripped by the P-Tile Avalon-ST IP
for PCIe.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

126
 UG-20225 | 2021.02.18

Send Feedback

6. Testbench
This chapter introduces the testbench for an Endpoint design example and a test
driver module. You can create this design example using design flows described in
Quick Start Guide chapter of the Intel FPGA P-Tile Avalon streaming IP for PCI Express
Design Example User Guide.

The testbench in this design example simulates up to a Gen4 x16 variant.

When configured as an Endpoint variation, the testbench instantiates a design

example with a P-Tile Endpoint and a Root Port BFM containing a second P-Tile
(configured as a Root Port) to interface with the Endpoint. The Root Port BFM provides
the following functions:
• A configuration routine that sets up all the basic configuration registers in the
Endpoint. This configuration allows the Endpoint application to be the target and
initiator of PCI Express transactions.
• A Verilog HDL procedure interface to initiate PCI Express transactions to the
Endpoint.

This testbench simulates the scenario of a single Root Port talking to a single
Endpoint.

The testbench uses a test driver module, altpcietb_bfm_rp_gen4_x16.sv, to

initiate the configuration and memory transactions. At startup, the test driver module
displays information from the Root Port and Endpoint Configuration Space registers, so
that you can correlate to the parameters you specified using the Parameter Editor.

Note: The Intel testbench and Root Port BFM provide a simple method to do basic testing of
the Application Layer logic that interfaces to the variation. This BFM allows you to
create and run simple task stimuli with configurable parameters to exercise basic
functionality of the Intel example design. The testbench and Root Port BFM are not
intended to be a substitute for a full verification environment. Corner cases and
certain traffic profile stimuli are not covered. Refer to the items listed below for further
details. To ensure the best verification coverage possible, Intel suggests strongly that
you obtain commercially available PCI Express verification IP and tools, in combination
with performing extensive hardware testing.

Intel Corporation. All rights reserved. Agilex, Altera, Arria, Cyclone, Enpirion, Intel, the Intel logo, MAX, Nios,
Quartus and Stratix words and logos are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or
other countries. Intel warrants performance of its FPGA and semiconductor products to current specifications in ISO
accordance with Intel's standard warranty, but reserves the right to make changes to any products and services 9001:2015
at any time without notice. Intel assumes no responsibility or liability arising out of the application or use of any Registered
information, product, or service described herein except as expressly agreed to in writing by Intel. Intel
customers are advised to obtain the latest version of device specifications before relying on any published
information and before placing orders for products or services.
*Other names and brands may be claimed as the property of others.
 6. Testbench
UG-20225 | 2021.02.18

Your Application Layer design may need to handle at least the following scenarios that
are not possible to create with the Intel testbench and the Root Port BFM, or are due
to the limitations of the example design:
• It is unable to generate or receive Vendor Defined Messages. Some systems
generate Vendor Defined Messages. The Hard IP block simply passes these
messages on to the Application Layer. Consequently, you should make the
decision, based on your application, whether to design the Application Layer to
process them.
• It can only handle received read requests that are less than or equal to the
currently set Maximum payload size option specified under the Device tab
under the PCI Express/PCI Capabilities GUI using the parameter editor. Many
systems are capable of handling larger read requests that are then returned in
multiple completions.
• It always returns a single completion for every read request. Some systems split
completions on every 64-byte address boundary.
• It always returns completions in the same order the read requests were issued.
Some systems generate the completions out-of-order.
• It is unable to generate zero-length read requests that some systems generate as
flush requests following some write transactions. The Application Layer must be
capable of generating the completions to the zero-length read requests.
• It uses fixed credit allocation.
• It does not support parity.
• It does not support multi-function designs.
• It incorrectly responds to Type 1 vendor-defined messages with CplD packets.

6.1. Endpoint Testbench

The example design and testbench are dynamically generated based on the
configuration that you choose for the P-Tile IP for PCIe. The testbench uses the
parameters that you specify in the Parameter Editor in Intel Quartus Prime.

This testbench simulates up to a ×16 PCI Express link using the serial PCI Express
interface. The testbench design does allow more than one PCI Express link to be
simulated at a time. The following figure presents a high level view of the design
example.

Figure 59. Design Example for Endpoint Designs

PCIe PIO Example Design Simulation Testbench

Root Port BFM PCIe PIO Example Design

(RP_BFM)

Avalon-ST Avalon-MM
Generated PCIe On-Chip
hip_serial data data
Endpoint PIO Application Memory
Variant (dut) (pio0) (MEM0)

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

128
6. Testbench
UG-20225 | 2021.02.18

The top-level of the testbench instantiates the following main modules:

• altpcietb_bfm_rp_gen4_x16.sv —This is the Root Port PCIe BFM.
//Directory path
<project_dir>/intel_pcie_ptile_ast_0_example_design/pcie_ed_tb/ip/
pcie_ed_tb/dut_pcie_tb_ip/intel_pcie_ptile_tbed_<ver>/sim

• pcie_ed_dut.ip: This is the Endpoint design with the parameters that you
specify.
//Directory path
<project_dir>/intel_pcie_ptile_ast_0_example_design/ip/pcie_ed

• pcie_ed_pio0.ip: This module is a target and initiator of transactions for the

PIO design example.
//Directory path
<project_dir>/intel_pcie_ptile_ast_0_example_design/ip/pcie_ed

• pcie_ed_sriov0.ip: This module is a target and initiator of transactions for the

SR-IOV design example.
//Directory path
<project_dir>/intel_pcie_ptile_ast_0_example_design/ip/pcie_ed

In addition, the testbench has routines that perform the following tasks:
• Generates the reference clock for the Endpoint at the required frequency.
• Provides a PCI Express reset at start up.

The SR-IOV design example testbench supports up to two Physical Functions (PFs) and
32 Virtual Functions (VFs) per PF.

For more details on the PIO design example testbench and SR-IOV design example
testbench, refer to the Intel FPGA P-Tile Avalon streaming IP for PCI Express Design
Example User Guide.

Note: By default, the serial_sim_hwtcl parameter in <project_dir>/

intel_pcie_ptile_ast_0_example_design/pcie_ed_tb/ip/pcie_ed_tb/
dut_pcie_tb_ip/intel_pcie_ptile_tbed_<ver>/sim/
intel_pcie_ptile_tbed_hwtcl.v is set to 1 for serial simulation. P-Tile does
not support parallel PIPE simulations.

Related Information
Intel FPGA P-Tile Avalon streaming IP for PCI Express Design Example User Guide

6.2. Test Driver Module

The test driver module, intel_pcie_ptile_tbed_hwtcl.v, instantiates the top-
level BFM,altpcietb_bfm_top_rp.v.

The top-level BFM completes the following tasks:

1. Instantiates the driver and monitor.
2. Instantiates the Root Port BFM.
3. Instantiates the serial interface.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

129
 6. Testbench
UG-20225 | 2021.02.18

The configuration module, altpcietb_g3bfm_configure.v, performs the following

tasks:
1. Configures and assigns the BARs.
2. Configures the Root Port and Endpoint.
3. Displays comprehensive Configuration Space, BAR, MSI, MSI-X, and AER settings.

6.3. Root Port BFM

The basic Root Port BFM provides a Verilog HDL task-based interface to request
transactions to issue on the PCI Express link. The Root Port BFM also handles requests
received from the PCI Express link. The following figure shows the major modules in
the Root Port BFM.

Figure 60. Root Port BFM

Root Port BFM

BFM Read/Write Shared Request Procedures

BFM Shared Memory (altpcietb_g3bfm_rdwr.v)
(altpcietb_g3bfm_
_shmem.v) BFM Configuration Procedures
(altpcietb_g3bfm_configure.v)
BFM Log Interface BFM Request Interface
(altpcietb_g3bfm_log.v) (altpcietb_g3bfm_req_intf.v)

Root Port Primary Drivers

Root BFM for Avalon-ST and Avalon-MM

Interfaces with or without Chaining DMA
(altpcietb_bfm_rp_gen3_x8.sv)

These modules implement the following functionality:

• BFM Log Interface, altpcietb_g3bfm_log.v and
altpcietb_bfm_rp_gen3_x8.sv: The BFM Log Interface provides routines for
writing commonly formatted messages to the simulator standard output and
optionally to a log file. It also provides controls that stop simulations on errors.
• BFM Read/Write Request Functions, altpcietb_bfm_rp_gen3_x8.sv: These
functions provide the basic BFM calls for PCI Express read and write requests.
• BFM Configuration Functions, altpcietb_g3bfm_configure.v : These
functions provide the BFM calls to request a configuration of the PCI Express link
and the Endpoint Configuration Space registers.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

130
6. Testbench
UG-20225 | 2021.02.18

• BFM shared memory, altpcietb_g3bfm_shmem.v: This module provides the

Root Port BFM shared memory. It implements the following functionality:
— Provides data for TX write operations
— Provides data for RX read operations
— Receives data for RX write operations
— Receives data for received completions
• BFM Request Interface, altpcietb_g3bfm_req_intf.v: This interface provides
the low-level interface between the altpcietb_g3bfm_rdwr and
altpcietb_g3bfm_configure procedures or functions and the Root Port RTL
Model. This interface stores a write-protected data structure containing the sizes
and values programmed in the BAR registers of the Endpoint. It also stores other
critical data used for internal BFM management.
• altpcietb_g3bfm_rdwr.v: This module contains the low-level read and write
tasks.
• Avalon-ST Interfaces, altpcietb_g3bfm_vc_intf_ast_common.v: These
interface modules handle the Root Port interface model. They take requests from
the BFM request interface and generate the required PCI Express transactions.
They handle completions received from the PCI Express link and notify the BFM
request interface when requests are complete. Additionally, they handle any
requests received from the PCI Express link, and store or fetch data from the
shared memory before generating the required completions.

In the PIO design example, the apps_type_hwtcl parameter is set to 3. The tests
run under this parameter value are defined in ebfm_cfg_rp_ep_rootport,
find_mem_bar and downstream_loop.

The function ebfm_cfg_rp_ep_rootport is described in

altpcietb_g3bfm_configure.v. This function performs the steps necessary to
configure the root port and the endpoint on the link. It includes:
• Root port memory allocation
• Root port configuration space (base limit, bus number, etc.)
• Endpoint configuration (BAR, Bus Master enable, maxpayload size, etc.)

The functions find_mem_bar and downstream_loop in

altpcietb_bfm_rp_gen3_x8.sv return the BAR implemented and perform the
memory Write and Read accesses to the BAR, respectively.

6.3.1. BFM Memory Map

The BFM shared memory is 2 MBs. The BFM shared memory maps to the first 2 MBs of
I/O space and also the first 2 MBs of memory space. When the Endpoint application
generates an I/O or memory transaction in this range, the BFM reads or writes the
shared memory.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

131
 6. Testbench
UG-20225 | 2021.02.18

6.3.2. Configuration Space Bus and Device Numbering

Enumeration assigns the Root Port interface device number 0 on internal bus number
0. Use the ebfm_cfg_rp_ep procedure to assign the Endpoint to any device number
on any bus number (greater than 0). The specified bus number is the secondary bus
in the Root Port Configuration Space.

6.3.3. Configuration of Root Port and Endpoint

Before you issue transactions to the Endpoint, you must configure the Root Port and
Endpoint Configuration Space registers.

The ebfm_cfg_rp_ep procedure in altpcietb_g3bfm_configure.v executes the

following steps to initialize the Configuration Space:
1. Sets the Root Port Configuration Space to enable the Root Port to send
transactions on the PCI Express link.
2. Sets the Root Port and Endpoint PCI Express Capability Device Control registers as
follows:
a. Disables Error Reporting in both the Root Port and Endpoint. The BFM
does not have error handling capability.
b. Enables Relaxed Ordering in both Root Port and Endpoint.
c. Enables Extended Tags for the Endpoint if the Endpoint has that capability.
d. Disables Phantom Functions, Aux Power PM, and No Snoop in both the
Root Port and Endpoint.
e. Sets the Max Payload Size to the value that the Endpoint supports because
the Root Port supports the maximum payload size.
f. Sets the Root Port Max Read Request Size to 4 KB because the example
Endpoint design supports breaking the read into as many completions as
necessary.
g. Sets the Endpoint Max Read Request Size equal to the Max Payload
Size because the Root Port does not support breaking the read request into
multiple completions.
3. Assigns values to all the Endpoint BAR registers. The BAR addresses are assigned
by the algorithm outlined below.
a. I/O BARs are assigned smallest to largest starting just above the ending
address of the BFM shared memory in I/O space and continuing as needed
throughout a full 32-bit I/O space.
b. The 32-bit non-prefetchable memory BARs are assigned smallest to largest,
starting just above the ending address of the BFM shared memory in memory
space and continuing as needed throughout a full 32-bit memory space.
c. The value of the addr_map_4GB_limit input to the ebfm_cfg_rp_ep
procedure controls the assignment of the 32-bit prefetchable and 64-bit
prefetchable memory BARS. The default value of the addr_map_4GB_limit
is 0.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

132
6. Testbench
UG-20225 | 2021.02.18

If the addr_map_4GB_limit input to the ebfm_cfg_rp_ep procedure is set

to 0, then the ebfm_cfg_rp_ep procedure assigns the 32-bit prefetchable
memory BARs largest to smallest, starting at the top of 32-bit memory space
and continuing as needed down to the ending address of the last 32-bit non-
prefetchable BAR.
However, if the addr_map_4GB_limit input is set to 1, the address map is
limited to 4 GB. The ebfm_cfg_rp_ep procedure assigns 32-bit and 64-bit
prefetchable memory BARs largest to smallest, starting at the top of the 32-bit
memory space and continuing as needed down to the ending address of the
last 32-bit non-prefetchable BAR.
d. If the addr_map_4GB_limit input to the ebfm_cfg_rp_ep procedure is set
to 0, then the ebfm_cfg_rp_ep procedure assigns the 64-bit prefetchable
memory BARs smallest to largest starting at the 4 GB address assigning
memory ascending above the 4 GB limit throughout the full 64-bit memory
space.
If the addr_map_4 GB_limit input to the ebfm_cfg_rp_ep procedure is set
to 1, the ebfm_cfg_rp_ep procedure assigns the 32-bit and the 64-bit
prefetchable memory BARs largest to smallest starting at the 4 GB address
and assigning memory by descending below the 4 GB address to memory
addresses as needed down to the ending address of the last 32-bit non-
prefetchable BAR.
The above algorithm cannot always assign values to all BARs when there are a
few very large (1 GB or greater) 32-bit BARs. Although assigning addresses to
all BARs may be possible, a more complex algorithm would be required to
effectively assign these addresses. However, such a configuration is unlikely to
be useful in real systems. If the procedure is unable to assign the BARs, it
displays an error message and stops the simulation.
4. Based on the above BAR assignments, the ebfm_cfg_rp_ep procedure assigns
the Root Port Configuration Space address windows to encompass the valid BAR
address ranges.
5. The ebfm_cfg_rp_ep procedure enables master transactions, memory address
decoding, and I/O address decoding in the Endpoint PCIe control register.

The ebfm_cfg_rp_ep procedure also sets up a bar_table data structure in BFM

shared memory that lists the sizes and assigned addresses of all Endpoint BARs. This
area of BFM shared memory is write-protected. Consequently, application logic write
accesses to this area cause a fatal simulation error.

BFM procedure calls to generate full PCIe addresses for read and write requests to
particular offsets from a BAR use this data structure. This procedure allows the
testbench code that accesses the Endpoint application logic to use offsets from a BAR
and avoid tracking specific addresses assigned to the BAR. The following table shows
how to use those offsets.

Table 105. BAR Table Structure

Offset (Bytes) Description

+0 PCI Express address in BAR0

+4 PCI Express address in BAR1

+8 PCI Express address in BAR2

continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

133
 6. Testbench
UG-20225 | 2021.02.18

Offset (Bytes) Description

+12 PCI Express address in BAR3

+16 PCI Express address in BAR4

+20 PCI Express address in BAR5

+24 PCI Express address in Expansion ROM BAR

+28 Reserved

+32 BAR0 read back value after being written with all 1’s (used to compute size)

+36 BAR1 read back value after being written with all 1’s

+40 BAR2 read back value after being written with all 1’s

+44 BAR3 read back value after being written with all 1’s

+48 BAR4 read back value after being written with all 1’s

+52 BAR5 read back value after being written with all 1’s

+56 Expansion ROM BAR read back value after being written with all 1’s

+60 Reserved

The configuration routine does not configure any advanced PCI Express capabilities
such as the AER capability.

Besides the ebfm_cfg_rp_ep procedure in altpcietb_bfm_rp_gen3_x8.sv,

routines to read and write Endpoint Configuration Space registers directly are available
in the Verilog HDL include file. After the ebfm_cfg_rp_ep procedure runs, the PCI
Express I/O and Memory Spaces have the layout shown in the following three figures.
The memory space layout depends on the value of the addr_map_4GB_limit input
parameter. The following figure shows the resulting memory space map when the
addr_map_4GB_limit is 1.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

134
6. Testbench
UG-20225 | 2021.02.18

Figure 61. Memory Space Layout—4 GB Limit

Address
0x0000 0000

Root Complex
Shared Memory

0x001F FF80
Configuration Scratch
Space Used by
BFM Routines - Not
Writeable by User
0x001F FFC0 Calls or Endpoint
BAR Table
Used by BFM
Routines - Not
Writeable by User
0x0020 0000 Calls or End Point

Endpoint Non-
Prefetchable Memory
Space BARs
Assigned Smallest
to Largest

Unused

Endpoint Memory
Space BARs
Prefetchable 32-bit
and 64-bit
Assigned Smallest
to Largest
0xFFFF FFFF

The following figure shows the resulting memory space map when the
addr_map_4GB_limit is 0.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

135
 6. Testbench
UG-20225 | 2021.02.18

Figure 62. Memory Space Layout—No Limit

Address
0x0000 0000

Root Complex
Shared Memory

0x001F FF80
Configuration Scratch
Space Used by
Routines - Not
Writeable by User
0x001F FF00 Calls or Endpoint
BAR Table
Used by BFM
Routines - Not
Writeable by User
0x0020 0000 Calls or Endpoint
Endpoint Non-
Prefetchable Memory
Space BARs
Assigned Smallest
BAR-Size Dependent to Largest

Unused
BAR-Size Dependent
Endpoint Memory
Space BARs
Prefetchable 32-bit
Assigned Smallest
0x0000 0001 0000 0000 to Largest
Endpoint Memory
Space BARs
Prefetchable 64-bit
Assigned Smallest
to Largest
BAR-Size Dependent

Unused

0xFFFF FFFF FFFF FFFF

The following figure shows the I/O address space.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

136
6. Testbench
UG-20225 | 2021.02.18

Figure 63. I/O Address Space

Address
0x0000 0000

Root Complex
Shared Memory

0x001F FF80
Configuration Scratch
Space Used by BFM
Routines - Not
Writeable by User
0x001F FFC0 Calls or Endpoint
BAR Table
Used by BFM
Routines - Not
Writeable by User
0x0020 0000 Calls or Endpoint

Endpoint
I/O Space BARs
Assigned Smallest
to Largest
BAR-Size Dependent

Unused

0xFFFF FFFF

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

137
 6. Testbench
UG-20225 | 2021.02.18

6.3.4. Issuing Read and Write Transactions to the Application Layer

The Root Port Application Layer issues read and write transactions by calling one of
the ebfm_bar procedures in altpcietb_g3bfm_rdwr.v. The procedures and
functions listed below are available in the Verilog HDL include file
altpcietb_g3bfm_rdwr.v. The complete list of available procedures and functions
is as follows:
• ebfm_barwr: writes data from BFM shared memory to an offset from a specific
Endpoint BAR. This procedure returns as soon as the request has been passed to
the VC interface module for transmission.
• ebfm_barwr_imm: writes a maximum of four bytes of immediate data (passed in
a procedure call) to an offset from a specific Endpoint BAR. This procedure returns
as soon as the request has been passed to the VC interface module for
transmission.
• ebfm_barrd_wait: reads data from an offset of a specific Endpoint BAR and
stores it in BFM shared memory. This procedure blocks waiting for the completion
data to be returned before returning control to the caller.
• ebfm_barrd_nowt: reads data from an offset of a specific Endpoint BAR and
stores it in the BFM shared memory. This procedure returns as soon as the request
has been passed to the VC interface module for transmission, allowing subsequent
reads to be issued in the interim.

These routines take as parameters a BAR number to access the memory space and
the BFM shared memory address of the bar_table data structure that was set up by
the ebfm_cfg_rp_ep procedure. (Refer to Configuration of Root Port and Endpoint.)
Using these parameters simplifies the BFM test driver routines that access an offset
from a specific BAR and eliminates calculating the addresses assigned to the specified
BAR.

The Root Port BFM does not support accesses to Endpoint I/O space BARs.

6.4. BFM Procedures and Functions

The BFM includes procedures, functions, and tasks to drive Endpoint application
testing. It also includes procedures to run the chaining DMA design example.

The BFM read and write procedures read and write data to BFM shared memory,
Endpoint BARs, and specified configuration registers. The procedures and functions are
available in the Verilog HDL. These procedures and functions support issuing memory
and configuration transactions on the PCI Express link.

6.4.1. ebfm_barwr Procedure

The ebfm_barwr procedure writes a block of data from BFM shared memory to an
offset from the specified Endpoint BAR. The length can be longer than the configured
MAXIMUM_PAYLOAD_SIZE. The procedure breaks the request up into multiple
transactions as needed. This routine returns as soon as the last transaction has been
accepted by the VC interface module.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

138
6. Testbench
UG-20225 | 2021.02.18

Location

Syntax ebfm_barwr(bar_table, bar_num, pcie_offset, lcladdr, byte_len, tclass)

Arguments bar_table Address of the Endpoint bar_table structure in BFM shared memory.
The bar_table structure stores the address assigned to each BAR so
that the driver code does not need to be aware of the actual assigned
addresses only the application specific offsets from the BAR.

bar_num Number of the BAR used with pcie_offset to determine PCI Express
address.

pcie_offset Address offset from the BAR base.

lcladdr BFM shared memory address of the data to be written.

byte_len Length, in bytes, of the data written. Can be 1 to the minimum of the
bytes remaining in the BAR space or BFM shared memory.

tclass Traffic class used for the PCI Express transaction.

6.4.2. ebfm_barwr_imm Procedure

The ebfm_barwr_imm procedure writes up to four bytes of data to an offset from the
specified Endpoint BAR.
Location

Syntax ebfm_barwr_imm(bar_table, bar_num, pcie_offset, imm_data, byte_len, tclass)

Arguments bar_table Address of the Endpoint bar_table structure in BFM shared memory.
The bar_table structure stores the address assigned to each BAR so
that the driver code does not need to be aware of the actual assigned
addresses only the application specific offsets from the BAR.

bar_num Number of the BAR used with pcie_offset to determine PCI Express
address.

pcie_offset Address offset from the BAR base.

imm_data Data to be written. In Verilog HDL, this argument is reg [31:0].In

both languages, the bits written depend on the length as follows:
Length Bits Written
• 4: 31 down to 0
• 3: 23 down to 0
• 2: 15 down to 0
• 1: 7 down to 0

byte_len Length of the data to be written in bytes. Maximum length is 4 bytes.

tclass Traffic class to be used for the PCI Express transaction.

6.4.3. ebfm_barrd_wait Procedure

The ebfm_barrd_wait procedure reads a block of data from the offset of the
specified Endpoint BAR and stores it in BFM shared memory. The length can be longer
than the configured maximum read request size; the procedure breaks the request up
into multiple transactions as needed. This procedure waits until all of the completion
data is returned and places it in shared memory.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

139
 6. Testbench
UG-20225 | 2021.02.18

Location

Syntax ebfm_barrd_wait(bar_table, bar_num, pcie_offset, lcladdr, byte_len, tclass)

Arguments bar_table Address of the Endpoint bar_table structure in BFM shared memory.
The bar_table structure stores the address assigned to each BAR so that
the driver code does not need to be aware of the actual assigned
addresses only the application specific offsets from the BAR.

bar_num Number of the BAR used with pcie_offset to determine PCI Express
address.

pcie_offset Address offset from the BAR base.

lcladdr BFM shared memory address where the read data is stored.

byte_len Length, in bytes, of the data to be read. Can be 1 to the minimum of

the bytes remaining in the BAR space or BFM shared memory.

tclass Traffic class used for the PCI Express transaction.

6.4.4. ebfm_barrd_nowt Procedure

The ebfm_barrd_nowt procedure reads a block of data from the offset of the
specified Endpoint BAR and stores the data in BFM shared memory. The length can be
longer than the configured maximum read request size; the procedure breaks the
request up into multiple transactions as needed. This routine returns as soon as the
last read transaction has been accepted by the VC interface module, allowing
subsequent reads to be issued immediately.
Location

Syntax ebfm_barrd_nowt(bar_table, bar_num, pcie_offset, lcladdr, byte_len, tclass)

Arguments bar_table Address of the Endpoint bar_table structure in BFM shared memory.

bar_num Number of the BAR used with pcie_offset to determine PCI Express
address.

pcie_offset Address offset from the BAR base.

lcladdr BFM shared memory address where the read data is stored.

byte_len Length, in bytes, of the data to be read. Can be 1 to the minimum of

the bytes remaining in the BAR space or BFM shared memory.

tclass Traffic Class to be used for the PCI Express transaction.

6.4.5. ebfm_cfgwr_imm_wait Procedure

The ebfm_cfgwr_imm_wait procedure writes up to four bytes of data to the
specified configuration register. This procedure waits until the write completion has
been returned.
Location

Syntax ebfm_cfgwr_imm_wait(bus_num, dev_num, fnc_num, imm_regb_ad, regb_ln, imm_data,

compl_status

Arguments bus_num PCI Express bus number of the target device.

dev_num PCI Express device number of the target device.

fnc_num Function number in the target device to be accessed.

continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

140
6. Testbench
UG-20225 | 2021.02.18

Location

regb_ad Byte-specific address of the register to be written.

regb_ln Length, in bytes, of the data written. Maximum length is four bytes. The
regb_ln and the regb_ad arguments cannot cross a DWORD
boundary.

imm_data Data to be written.

This argument is reg [31:0].
The bits written depend on the length:
• 4: 31 down to 0
• 3: 23 down to 0
• 2: 15 down to 0
• 1: 7 down to 0

compl_status This argument is reg [2:0].

This argument is the completion status as specified in the PCI Express
specification. The following encodings are defined:
• 3’b000: SC— Successful completion
• 3’b001: UR— Unsupported Request
• 3’b010: CRS — Configuration Request Retry Status
• 3’b100: CA — Completer Abort

6.4.6. ebfm_cfgwr_imm_nowt Procedure

The ebfm_cfgwr_imm_nowt procedure writes up to four bytes of data to the
specified configuration register. This procedure returns as soon as the VC interface
module accepts the transaction, allowing other writes to be issued in the interim. Use
this procedure only when successful completion status is expected.
Location

Syntax ebfm_cfgwr_imm_nowt(bus_num, dev_num, fnc_num, imm_regb_adr, regb_len,

imm_data)

Arguments bus_num PCI Express bus number of the target device.

dev_num PCI Express device number of the target device.

fnc_num Function number in the target device to be accessed.

regb_ad Byte-specific address of the register to be written.

regb_ln Length, in bytes, of the data written. Maximum length is four bytes, The
regb_ln the regb_ad arguments cannot cross a DWORD boundary.

imm_data Data to be written

This argument is reg [31:0].
In both languages, the bits written depend on the length. The following
encodes are defined.
• 4: [31:0]
• 3: [23:0]
• 2: [15:0]
• 1: [7:0]

6.4.7. ebfm_cfgrd_wait Procedure

The ebfm_cfgrd_wait procedure reads up to four bytes of data from the specified
configuration register and stores the data in BFM shared memory. This procedure
waits until the read completion has been returned.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

141
 6. Testbench
UG-20225 | 2021.02.18

Location

Syntax ebfm_cfgrd_wait(bus_num, dev_num, fnc_num, regb_ad, regb_ln, lcladdr,

compl_status)

Arguments bus_num PCI Express bus number of the target device.

dev_num PCI Express device number of the target device.

fnc_num Function number in the target device to be accessed.

regb_ad Byte-specific address of the register to be written.

regb_ln Length, in bytes, of the data read. Maximum length is four bytes. The
regb_ln and the regb_ad arguments cannot cross a DWORD
boundary.

lcladdr BFM shared memory address of where the read data should be placed.

compl_status Completion status for the configuration transaction.

This argument is reg [2:0].
In both languages, this is the completion status as specified in the PCI
Express specification. The following encodings are defined.
• 3’b000: SC— Successful completion
• 3’b001: UR— Unsupported Request
• 3’b010: CRS — Configuration Request Retry Status
• 3’b100: CA — Completer Abort

6.4.8. ebfm_cfgrd_nowt Procedure

The ebfm_cfgrd_nowt procedure reads up to four bytes of data from the specified
configuration register and stores the data in the BFM shared memory. This procedure
returns as soon as the VC interface module has accepted the transaction, allowing
other reads to be issued in the interim. Use this procedure only when successful
completion status is expected and a subsequent read or write with a wait can be used
to guarantee the completion of this operation.
Location

Syntax ebfm_cfgrd_nowt(bus_num, dev_num, fnc_num, regb_ad, regb_ln, lcladdr)

Arguments bus_num PCI Express bus number of the target device.

dev_num PCI Express device number of the target device.

fnc_num Function number in the target device to be accessed.

regb_ad Byte-specific address of the register to be written.

regb_ln Length, in bytes, of the data written. Maximum length is four bytes. The
regb_ln and regb_ad arguments cannot cross a DWORD boundary.

lcladdr BFM shared memory address where the read data should be placed.

6.4.9. BFM Configuration Procedures

All Verilog HDL arguments are type integer and are input-only unless specified
otherwise.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

142
6. Testbench
UG-20225 | 2021.02.18

6.4.9.1. ebfm_cfg_rp_ep Procedure

The ebfm_cfg_rp_ep procedure configures the Root Port and Endpoint Configuration
Space registers for operation.
Location

Syntax ebfm_cfg_rp_ep(bar_table, ep_bus_num, ep_dev_num, rp_max_rd_req_size,

display_ep_config, addr_map_4GB_limit)

Arguments bar_table Address of the Endpoint bar_table structure in BFM shared memory.
This routine populates the bar_table structure. The bar_table
structure stores the size of each BAR and the address values assigned to
each BAR. The address of the bar_table structure is passed to all
subsequent read and write procedure calls that access an offset from a
particular BAR.

ep_bus_num PCI Express bus number of the target device. This number can be any
value greater than 0. The Root Port uses this as the secondary bus
number.

ep_dev_num PCI Express device number of the target device. This number can be
any value. The Endpoint is automatically assigned this value when it
receives the first configuration transaction.

rp_max_rd_req_size Maximum read request size in bytes for reads issued by the Root Port.
This parameter must be set to the maximum value supported by the
Endpoint Application Layer. If the Application Layer only supports reads
of the MAXIMUM_PAYLOAD_SIZE, then this can be set to 0 and the read
request size is set to the maximum payload size. Valid values for this
argument are 0, 128, 256, 512, 1,024, 2,048 and 4,096.

display_ep_config When set to 1 many of the Endpoint Configuration Space registers are
displayed after they have been initialized, causing some additional reads
of registers that are not normally accessed during the configuration
process such as the Device ID and Vendor ID.

addr_map_4GB_limit When set to 1 the address map of the simulation system is limited to 4
GB. Any 64-bit BARs are assigned below the 4 GB limit.

6.4.9.2. ebfm_cfg_decode_bar Procedure

The ebfm_cfg_decode_bar procedure analyzes the information in the BAR table for
the specified BAR and returns details about the BAR attributes.
Location

Syntax ebfm_cfg_decode_bar(bar_table, bar_num, log2_size, is_mem, is_pref, is_64b)

Arguments bar_table Address of the Endpoint bar_table structure in BFM shared memory.

bar_num BAR number to analyze.

log2_size This argument is set by the procedure to the log base 2 of the size of
the BAR. If the BAR is not enabled, this argument is set to 0.

is_mem The procedure sets this argument to indicate if the BAR is a memory
space BAR (1) or I/O Space BAR (0).

is_pref The procedure sets this argument to indicate if the BAR is a prefetchable
BAR (1) or non-prefetchable BAR (0).

is_64b The procedure sets this argument to indicate if the BAR is a 64-bit BAR
(1) or 32-bit BAR (0). This is set to 1 only for the lower numbered BAR
of the pair.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

143
 6. Testbench
UG-20225 | 2021.02.18

6.4.10. BFM Shared Memory Access Procedures

These procedures and functions support accessing the BFM shared memory.

6.4.10.1. Shared Memory Constants

Table 106. Constants: Verilog HDL Type INTEGER

Constant Description

SHMEM_FILL_ZEROS Specifies a data pattern of all zeros

SHMEM_FILL_BYTE_INC Specifies a data pattern of incrementing 8-bit bytes (0x00, 0x01, 0x02,
etc.)

SHMEM_FILL_WORD_INC Specifies a data pattern of incrementing 16-bit words (0x0000, 0x0001,

0x0002, etc.)

SHMEM_FILL_DWORD_INC Specifies a data pattern of incrementing 32-bit DWORDs (0x00000000,

0x00000001, 0x00000002, etc.)

SHMEM_FILL_QWORD_INC Specifies a data pattern of incrementing 64-bit qwords

(0x0000000000000000, 0x0000000000000001, 0x0000000000000002,
etc.)

SHMEM_FILL_ONE Specifies a data pattern of all ones

6.4.10.2. shmem_write Task

The shmem_write procedure writes data to the BFM shared memory.

Location

Syntax shmem_write(addr, data, leng)

Arguments addr BFM shared memory starting address for writing data

data Data to write to BFM shared memory.

This parameter is implemented as a 64-bit vector. leng is 1–8 bytes.
Bits 7 down to 0 are written to the location specified by addr; bits 15
down to 8 are written to the addr+1 location, etc.

length Length, in bytes, of data written

6.4.10.3. shmem_read Function

The shmem_read function reads data to the BFM shared memory.

Location

Syntax data:= shmem_read(addr, leng)

Arguments addr BFM shared memory starting address for reading data

leng Length, in bytes, of data read

Return data Data read from BFM shared memory.

This parameter is implemented as a 64-bit vector. leng is 1- 8 bytes. If
leng is less than 8 bytes, only the corresponding least significant bits of
the returned data are valid.
Bits 7 down to 0 are read from the location specified by addr; bits 15
down to 8 are read from the addr+1 location, etc.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

144
6. Testbench
UG-20225 | 2021.02.18

6.4.10.4. shmem_display Verilog HDL Function

The shmem_display Verilog HDL function displays a block of data from the BFM
shared memory.
Location

Syntax Verilog HDL: dummy_return:=shmem_display(addr, leng, word_size, flag_addr,

msg_type);

Arguments addr BFM shared memory starting address for displaying data.

leng Length, in bytes, of data to display.

word_size Size of the words to display. Groups individual bytes into words. Valid
values are 1, 2, 4, and 8.

flag_addr Adds a <== flag to the end of the display line containing this address.
Useful for marking specific data. Set to a value greater than 2**21 (size
of BFM shared memory) to suppress the flag.

msg_type Specifies the message type to be displayed at the beginning of each

line. See “BFM Log and Message Procedures” on page 18–37 for more
information about message types. Set to one of the constants defined in
Table 18–36 on page 18–41.

6.4.10.5. shmem_fill Procedure

The shmem_fill procedure fills a block of BFM shared memory with a specified data
pattern.
Location

Syntax shmem_fill(addr, mode, leng, init)

Arguments addr BFM shared memory starting address for filling data.

mode Data pattern used for filling the data. Should be one of the constants
defined in section Shared Memory Constants.

leng Length, in bytes, of data to fill. If the length is not a multiple of the
incrementing data pattern width, then the last data pattern is truncated
to fit.

init Initial data value used for incrementing data pattern modes. This
argument is reg [63:0].
The necessary least significant bits are used for the data patterns that
are smaller than 64 bits.

6.4.10.6. shmem_chk_ok Function

The shmem_chk_ok function checks a block of BFM shared memory against a

specified data pattern.
Location

Syntax result:= shmem_chk_ok(addr, mode, leng, init, display_error)

Arguments addr BFM shared memory starting address for checking data.

mode Data pattern used for checking the data. Should be one of the constants
defined in section “Shared Memory Constants” on page 18–35.

leng Length, in bytes, of data to check.

continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

145
 6. Testbench
UG-20225 | 2021.02.18

Location

init This argument is reg [63:0].The necessary least significant bits are
used for the data patterns that are smaller than 64-bits.

display_error When set to 1, this argument displays the data failing comparison on
the simulator standard output.

Return Result Result is 1-bit.

• 1’b1 — Data patterns compared successfully
• 1’b0 — Data patterns did not compare successfully

6.4.11. BFM Log and Message Procedures

These procedures provide support for displaying messages in a common format,
suppressing informational messages, and stopping simulation on specific message
types.

The following constants define the type of message and their values determine
whether a message is displayed or simulation is stopped after a specific message.
Each displayed message has a specific prefix, based on the message type in the
following table.

You can suppress the display of certain message types. The default values determining
whether a message type is displayed are defined in the following table. To change the
default message display, modify the display default value with a procedure call to
ebfm_log_set_suppressed_msg_mask.

Certain message types also stop simulation after the message is displayed. The
following table shows the default value determining whether a message type stops
simulation. You can specify whether simulation stops for particular messages with the
procedure ebfm_log_set_stop_on_msg_mask.

All of these log message constants type integer.

Table 107. Log Messages

Constant Description Mask Bit Display Simulation Message
(Message No by Default Stops by Prefix
Type) Default

EBFM_MSG_D Specifies debug messages. 0 No No DEBUG:

EBUG

EBFM_MSG_I Specifies informational messages, 1 Yes No INFO:

NFO such as configuration register
values, starting and ending of tests.

EBFM_MSG_W Specifies warning messages, such 2 Yes No WARNING:

ARNING as tests being skipped due to the
specific configuration.

EBFM_MSG_E Specifies additional information for 3 Yes No ERROR:

RROR_INFO an error. Use this message to
display preliminary information
before an error message that stops
simulation.
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

146
6. Testbench
UG-20225 | 2021.02.18

Constant Description Mask Bit Display Simulation Message

(Message No by Default Stops by Prefix
Type) Default

EBFM_MSG_E Specifies a recoverable error that 4 Yes No ERROR:

RROR_CONTI allows simulation to continue. Use
NUE this error for data comparison
failures.

EBFM_MSG_E Specifies an error that stops N/A Yes Yes FATAL:

RROR_FATAL simulation because the error leaves Cannot Cannot suppress
the testbench in a state where suppress
further simulation is not possible.

EBFM_MSG_E Used for BFM test driver or Root N/A Y Y FATAL:

RROR_FATAL Port BFM fatal errors. Specifies an Cannot Cannot suppress
_TB_ERR error that stops simulation because suppress
the error leaves the testbench in a
state where further simulation is
not possible. Use this error
message for errors that occur due
to a problem in the BFM test driver
module or the Root Port BFM, that
are not caused by the Endpoint
Application Layer being tested.

6.4.11.1. ebfm_display Verilog HDL Function

The ebfm_display procedure or function displays a message of the specified type to

the simulation standard output and also the log file if ebfm_log_open is called.

A message can be suppressed, simulation can be stopped or both based on the default
settings of the message type and the value of the bit mask when each of the
procedures listed below is called. You can call one or both of these procedures based
on what messages you want displayed and whether or not you want simulation to stop
for specific messages.
• When ebfm_log_set_suppressed_msg_mask is called, the display of the
message might be suppressed based on the value of the bit mask.
• When ebfm_log_set_stop_on_msg_mask is called, the simulation can be
stopped after the message is displayed, based on the value of the bit mask.

Location

Syntax Verilog HDL: dummy_return:=ebfm_display(msg_type, message);

Argument msg_type Message type for the message. Should be one of the constants defined
in Table 106 on page 144.

message The message string is limited to a maximum of 100 characters. Also,

because Verilog HDL does not allow variable length strings, this routine
strips off leading characters of 8’h00 before displaying the message.

Return always 0 Applies only to the Verilog HDL routine.

6.4.11.2. ebfm_log_stop_sim Verilog HDL Function

The ebfm_log_stop_sim procedure stops the simulation.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

147
 6. Testbench
UG-20225 | 2021.02.18

Location

Syntax Verilog HDL: return:=ebfm_log_stop_sim(success);

Argument success When set to a 1, this process stops the simulation with a message
indicating successful completion. The message is prefixed with
SUCCESS.
Otherwise, this process stops the simulation with a message indicating
unsuccessful completion. The message is prefixed with FAILURE.

Return Always 0 This value applies only to the Verilog HDL function.

6.4.11.3. ebfm_log_set_suppressed_msg_mask Task

The ebfm_log_set_suppressed_msg_mask procedure controls which message

types are suppressed.
Location

Syntax ebfm_log_set_suppressed_msg_mask (msg_mask)

Argument msg_mask This argument is reg [EBFM_MSG_ERROR_CONTINUE:

EBFM_MSG_DEBUG].
A 1 in a specific bit position of the msg_mask causes messages of the
type corresponding to the bit position to be suppressed.

6.4.11.4. ebfm_log_set_stop_on_msg_mask Verilog HDL Task

The ebfm_log_set_stop_on_msg_mask procedure controls which message types

stop simulation. This procedure alters the default behavior of the simulation when
errors occur as described in the BFM Log and Message Procedures.
Location

Syntax ebfm_log_set_stop_on_msg_mask (msg_mask)

Argument msg_mask This argument is reg

[EBFM_MSG_ERROR_CONTINUE:EBFM_MSG_DEBUG].
A 1 in a specific bit position of the msg_mask causes messages of the
type corresponding to the bit position to stop the simulation after the
message is displayed.

6.4.11.5. ebfm_log_open Verilog HDL Function

The ebfm_log_open procedure opens a log file of the specified name. All displayed
messages are called by ebfm_display and are written to this log file as simulator
standard output.
Location

Syntax ebfm_log_open (fn)

Argument fn This argument is type string and provides the file name of log file to
be opened.

6.4.11.6. ebfm_log_close Verilog HDL Function

The ebfm_log_close procedure closes the log file opened by a previous call to
ebfm_log_open.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

148
6. Testbench
UG-20225 | 2021.02.18

Location

Syntax ebfm_log_close

Argument NONE

6.4.12. Verilog HDL Formatting Functions

6.4.12.1. himage1

This function creates a one-digit hexadecimal string representation of the input

argument that can be concatenated into a larger message string and passed to
ebfm_display.
Location

Syntax string:= himage(vec)

Argument vec Input data type reg with a range of 3:0.

Return range string Returns a 1-digit hexadecimal representation of the input argument.
Return data is type reg with a range of 8:1

6.4.12.2. himage2

This function creates a two-digit hexadecimal string representation of the input

argument that can be concatenated into a larger message string and passed to
ebfm_display.
Location

Syntax string:= himage(vec)

Argument range vec Input data type reg with a range of 7:0.

Return range string Returns a 2-digit hexadecimal presentation of the input argument,
padded with leading 0s, if they are needed. Return data is type reg with
a range of 16:1

6.4.12.3. himage4

This function creates a four-digit hexadecimal string representation of the input

argument can be concatenated into a larger message string and passed to
ebfm_display.
Location

Syntax string:= himage(vec)

Argument range vec Input data type reg with a range of 15:0.

Return range Returns a four-digit hexadecimal representation of the input argument, padded with leading 0s, if they
are needed. Return data is type reg with a range of 32:1.

6.4.12.4. himage8

This function creates an 8-digit hexadecimal string representation of the input

argument that can be concatenated into a larger message string and passed to
ebfm_display.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

149
 6. Testbench
UG-20225 | 2021.02.18

Location

Syntax string:= himage(vec)

Argument range vec Input data type reg with a range of 31:0.

Return range string Returns an 8-digit hexadecimal representation of the input argument,
padded with leading 0s, if they are needed. Return data is type reg with
a range of 64:1.

6.4.12.5. himage16

This function creates a 16-digit hexadecimal string representation of the input

argument that can be concatenated into a larger message string and passed to
ebfm_display.
Location

Syntax string:= himage(vec)

Argument range vec Input data type reg with a range of 63:0.

Return range string Returns a 16-digit hexadecimal representation of the input argument,
padded with leading 0s, if they are needed. Return data is type reg with
a range of 128:1.

6.4.12.6. dimage1

This function creates a one-digit decimal string representation of the input argument
that can be concatenated into a larger message string and passed to ebfm_display.
Location

Syntax string:= dimage(vec)

Argument range vec Input data type reg with a range of 31:0.

Return range string Returns a 1-digit decimal representation of the input argument that is
padded with leading 0s if necessary. Return data is type reg with a
range of 8:1.
Returns the letter U if the value cannot be represented.

6.4.12.7. dimage2

This function creates a two-digit decimal string representation of the input argument
that can be concatenated into a larger message string and passed to ebfm_display.
Location

Syntax string:= dimage(vec)

Argument range vec Input data type reg with a range of 31:0.

Return range string Returns a 2-digit decimal representation of the input argument that is
padded with leading 0s if necessary. Return data is type reg with a
range of 16:1.
Returns the letter U if the value cannot be represented.

6.4.12.8. dimage3

This function creates a three-digit decimal string representation of the input argument
that can be concatenated into a larger message string and passed to ebfm_display.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

150
6. Testbench
UG-20225 | 2021.02.18

Location

Syntax string:= dimage(vec)

Argument range vec Input data type reg with a range of 31:0.

Return range string Returns a 3-digit decimal representation of the input argument that is
padded with leading 0s if necessary. Return data is type reg with a
range of 24:1.
Returns the letter U if the value cannot be represented.

6.4.12.9. dimage4

This function creates a four-digit decimal string representation of the input argument
that can be concatenated into a larger message string and passed to ebfm_display.
Location

Syntax string:= dimage(vec)

Argument range vec Input data type reg with a range of 31:0.

Return range string Returns a 4-digit decimal representation of the input argument that is
padded with leading 0s if necessary. Return data is type reg with a
range of 32:1.
Returns the letter U if the value cannot be represented.

6.4.12.10. dimage5

This function creates a five-digit decimal string representation of the input argument
that can be concatenated into a larger message string and passed to ebfm_display.
Location

Syntax string:= dimage(vec)

Argument range vec Input data type reg with a range of 31:0.

Return range string Returns a 5-digit decimal representation of the input argument that is
padded with leading 0s if necessary. Return data is type reg with a
range of 40:1.
Returns the letter U if the value cannot be represented.

6.4.12.11. dimage6

This function creates a six-digit decimal string representation of the input argument
that can be concatenated into a larger message string and passed to ebfm_display.
Location

Syntax string:= dimage(vec)

Argument range vec Input data type reg with a range of 31:0.

Return range string Returns a 6-digit decimal representation of the input argument that is
padded with leading 0s if necessary. Return data is type reg with a
range of 48:1.
Returns the letter U if the value cannot be represented.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

151
 6. Testbench
UG-20225 | 2021.02.18

6.4.12.12. dimage7

This function creates a seven-digit decimal string representation of the input argument
that can be concatenated into a larger message string and passed to ebfm_display.
Location

Syntax string:= dimage(vec)

Argument range vec Input data type reg with a range of 31:0.

Return range string Returns a 7-digit decimal representation of the input argument that is
padded with leading 0s if necessary. Return data is type reg with a
range of 56:1.
Returns the letter <U> if the value cannot be represented.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

152
 UG-20225 | 2021.02.18

Send Feedback

7. Troubleshooting/Debugging
As you bring up your PCI Express system, you may face issues related to FPGA
configuration, link training, BIOS enumeration, data transfer, and so on. This chapter
suggests some strategies to resolve the common issues that occur during bring-up.

You can additionally use the P-Tile Debug Toolkit to identify the issues.

7.1. Hardware
Typically, PCI Express link-up involves the following steps:
1. Link training
2. BIOS enumeration and data transfer

The following sections describe the flow to debug link issues during the hardware
bring-up. Intel recommends a systematic approach to diagnosing issues as illustrated
in the following figure.

Additionally, you can use the P-Tile Debug Toolkit for debugging the PCIe links when
using the P-Tile Avalon-ST IP for PCI Express. The P-Tile Debug Toolkit includes the
following features:
• Protocol and link status information.
• Basic and advanced debugging capabilities including PMA register access and Eye
viewing capability.

Intel Corporation. All rights reserved. Agilex, Altera, Arria, Cyclone, Enpirion, Intel, the Intel logo, MAX, Nios,
Quartus and Stratix words and logos are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or
other countries. Intel warrants performance of its FPGA and semiconductor products to current specifications in ISO
accordance with Intel's standard warranty, but reserves the right to make changes to any products and services 9001:2015
at any time without notice. Intel assumes no responsibility or liability arising out of the application or use of any Registered
information, product, or service described herein except as expressly agreed to in writing by Intel. Intel
customers are advised to obtain the latest version of device specifications before relying on any published
information and before placing orders for products or services.
*Other names and brands may be claimed as the property of others.
 7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

Figure 64. PCI Express Debug Flow Chart

Is the link
Go to “7.1.1 Debugging
Start training Link training issues”
successful? No
System Reset

Yes

Is the data Go to “7.1.2 Debugging data

transfer transfer and
successful? No performance issues”

Yes

End

7.1.1. Debugging Link Training Issues

The Physical Layer automatically performs link training and initialization without
software intervention. This is a well-defined process to configure and initialize the
device's Physical Layer and link so that PCIe packets can be transmitted.

Some examples of link training issues include:

• Link fails to negotiate to expected link speed.
• Link fails to negotiate to the expected link width.
• LTSSM fails to reach/stay stable at L0.

Flow Chart for Debugging Link Training Issues

Use the flow chart below to identify the potential cause of the issue seen during link
training when using the P-Tile Avalon-ST IP for PCI Express.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

154
7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

Figure 65. Link Training Debugging Flow

Begin

A.Observation: Frequent transitions on ltssm_state_o

Does link signal between L0 and Recovery.rcvrlock states
Does the go to L0 at
LTSSM advertised speed with Issue: Signal Integrity issues (or) sub optimal EQ settings
enter L0? Yes frequent recoveries? Yes Resolution: Redo the Equalization (*)

No A.Observation: Wrong lane numbers encoded in

TS1/TS2 (Observed using Protocol Analyzer)
Issue: Improper lane reversal
Resolution: Check the lane routing
Does the
link train to OR
No L0 with reduced B.Observation: Timeout during EQ Phases on few
lane width? Yes lanes when monitoring ltssm_state_o signal
Issue: Signal Integrity issues/Sub optimal EQ settings
on few lanes
Resolution: Redo the Equalization (*)

No

A.Observation: Loop of Detect.Quiet

–> Detect.Active –> Polling.Active
–> Recovery.rcvrlock transitions observed on
Does the link ltssm_state_o signal
End train to L0 at Issue: Poor refclk quality
No reduced speed? Yes Resolution: Check the reference clock quality is good.
(e.g. Jitter, phase noise, etc). Ensure that the clocks
used are in accordance with the guidelines described in
the User Guide

OR
B.Observation: Timeout during EQ Phases on few
lanes when monitoring ltssm_state_o signal
Is the link Yes Is there
receiver detected Issue: Signal Integrity issues/Sub optimal EQ settings
out of
at the far end? on few lanes
reset?
Resolution: Redo the Equalization (*)

Yes
No No

Observation: ltssm_state_o signal Observation: ltssm_state_o signal toggles Observation: ltssm_state_o signal
stuck at Detect.Quiet state between Detect.Quiet and Detect.Active. Check the transitions from Detect.Quiet –>
Receiver detection status from the registers for Detect.Active –> Polling.Active –>
Issue: IP is in reset state
successful receiver detection Polling.Compliance states.
Resolution: Check if the pin_perst_n
Issue: Far end receiver not detected by the FPGA TX Issue: Far end device failing receiver detection
reset signal is in reset
Resolution: Check coupling capacitance, Resolution: Check far end coupling capacitance,
far end termination resistance and TX OCT values are near end termination resistance and TX OCT values
in accordance to the spec are in accordance to the spec

Note: (*) Redo the equalization using the Link Equalization Request 8.0 GT/s bit
of the Link Status 2 register for 8.0 GT/s or Link Equalization Request 16.0 GT/s
bit of the 16.0 GT/s Status Register.

Use the following debug tools for debugging link training issues observed on the PCI
Express link when using the P-tile Avalon-ST IP for PCI Express.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

155
 7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

7.1.1.1. Generic Tools and Utilities

You can use utilities like lspci, setpci to obtain general information of the device like
link speed, link width etc.

Example: To read the negotiated link speed for the P-Tile device in a system, you can
use the following commands:

sudo lspci –s $bdf -vvv

-s refers to “slot” and is used with the bus/device/function number (bdf) information.
Use this command if you know the bdf of the device in the system topology.

sudo lspci –d <1172>:$did -vvv

-d refers to device and is used with the device ID (vid:did). Use this command to
search using the device ID.

Figure 66. lspci Output

The LnkCap under Capabilities indicates the advertised link speed and width
capabilities of the device. The LnkSta under Capabilities indicates the negotiated
link speed and width of the device.

7.1.1.2. SignalTapII Logic Analyzer

Using the SignalTapII Logic Analyzer, you can monitor the following top-level signals
from the P-Tile Avalon-ST IP for PCI Express to confirm the failure symptom for any
port type (Root Port, Endpoint or TLP Bypass) and configuration (Gen4/Gen3).

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

156
7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

Table 108. Top-Level Signals to be Monitored for Debugging

Signals Description Expected Value for Successful Link-
up

p<n>_pin_perst_n where n = 0, 1, Active-low asynchronous output signal 1'b1

2, 3 from the PCIe Hard IP. It is derived
from the pin_perst_n input signal.

p<n>_reset_status_n where n = 0, Active-low output signal from the PCIe 1'b1

1, 2, 3 Hard IP, synchronous to
coreclkout_hip.
Held low until pin_perst_n is
deasserted and the PCIe Hard IP
comes out of reset, synchronous to
coreclkout_hip.
When port bifurcation is used, there is
one such signal for each Avalon-ST
interface.

ninit_done Active-low output signal from the Reset 1'b0

Release Intel FPGA IP. High indicates
that the FPGA device is not yet fully
configured, and low indicates the
device has been configured and is in
normal operating mode.
For more details on the Reset Release
Intel FPGA IP, refer to https://
www.intel.com/content/www/us/en/
programmable/documentation/
prh1555609801770.html

link_up_o Active-high output signal from the PCIe 1'b1

Hard IP, synchronous to
coreclkout_hip.
Indicates that the Physical Layer link is
up.

dl_up_o Active-high output signal from the PCIe 1'b1

Hard IP, synchronous to
coreclkout_hip.
Indicates that the Data Link Layer is
active.

ltssm_state_o[5:0] Indicates the LTSSM state, 6'h11 (L0)

synchronous to coreclkout_hip.

Negotiated link speed using the Use the Transaction Layer tl_cfg_add_o[4:0] = 5'h05
Transaction Layer Configuration Output Configuration Output interface (tl_cfg) tl_cfg_ctl_o[15:12] =
interface (tl_cfg): to monitor the auto-negotiated link
speed. • 4’h01 (Gen1)
tl_cfg_add_o[4:0]
• 4’h02 (Gen2)
tl_cfg_ctl_o[15:12]
• 4’h04 (Gen3)
tl_cfg_func_o[2:0] • 4’h08 (Gen4)
tl_cfg_func_o[2:0] (NA for x4) =
• 3’b000: PF0
• 3'b001: PF1, etc.

Negotiated link width using the Use the Transaction Layer tl_cfg_add_o[4:0] = 5'h1C
Transaction Layer Configuration Output Configuration Output interface (tl_cfg) tl_cfg_ctl_o[5:0] =
interface (tl_cfg): to monitor the auto-negotiated link
width. • 6’h01 (x1)
tl_cfg_add_o[4:0]
• 6’h02 (x2)
tl_cfg_ctl_o[15:12]
• 6’h04 (x4)
tl_cfg_func_o[2:0] • 6’h08 (x8)
• 6'h10 (x16)

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

157
 7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

7.1.1.3. Additional Debug Tools

Use the Hard IP reconfiguration interface and PHY reconfiguration interface on the P-
Tile Avalon-ST IP for PCI Express to access additional registers (for example, receiver
detection, lane reversal etc.).

Figure 67. Register Access for Debug

PHY
PCIe Controllers
Port N
PLLA PMA x16
PCIe PCIe MAC DLL TL Hard IP Reconfig
PLLB Quad N
x16 Lanes PCS Interface
PHY Registers
Registers

PHY Reconfig
Interface

Using the Hard IP Reconfiguration Interface

Refer to the section Hard IP Reconfiguration Interface for details on this interface and
the associated address map.

The following table lists the address offsets and bit settings for the PHY status
registers. Use the Hard IP Reconfiguration Interface to access these read-only
registers.

Table 109. Hard IP Reconfiguration Interface Register Map for PHY Status
Offset Bit Position Register

0x0003E9 [0] RX polarity

[1] RX detection

[2] RX Valid

[3] RX Electrical Idle

[4] TX Electrical Idle

0x0003EC [7] Framing error

0x0003ED [7] Lane reversal

Follow the steps below to access registers in Table 109 on page 158 using the Hard IP
reconfiguration interface:
1. Enable the Hard IP reconfiguration interface (User Avalon-MM interface) using the
IP Parameter Editor.
2. Set the lane number for which you want to read the status by performing a read-
modify-write to the address hip_reconfig_addr_i[20:0] with write data of
lane number on hip_reconfig_writedata_i[7:0] using the Hard IP
reconfiguration interface signals.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

158
7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

• hip_reconfig_write_i = 1’b1
• hip_reconfig_addr_i[20:0] = 0x0003E8
• hip_reconfig_writedata_i[3:0] = <Lane number>, where Lane number
= 4’h0 for lane 0, 4’h1 for lane 1, 4’h2 for lane 2, …
3. Read the status of the register you want by performing a read operation from the
address hip_reconfig_addr_i[20:0] using the Hard IP reconfiguration
interface signals.
• hip_reconfig_read_i = 1’b1
• hip_reconfig_addr_i[20:0] = <offset>
Offset = Refer to Table 109 on page 158 for the offset mapping.
• hip_reconfig_readdata_o[7:0] = Refer to Table 109 on page 158 for the
bit position mapping.

Example 1: To read the RX detection status of Lane0 using the registers

1. Enable the Hard IP reconfiguration interface using the IP Parameter Editor.
2. Perform read-modify-write to address 0x0003E8 to set the lane number to 0 using
the Hard IP reconfiguration interface signals.
• hip_reconfig_write_i = 1’b1
• hip_reconfig_addr_i[20:0] = 0x0003E8
• hip_reconfig_writedata_i[3:0] = 4'h0
3. Read the status of the RX detection register by performing a read operation from
the address 0x0003E9[1] using the Hard IP reconfiguration interface signals.
• hip_reconfig_read_i = 1’b1
• hip_reconfig_addr_i[20:0] = 0x0003E9
• hip_reconfig_readdata_o[1] = 1'b1 (Far end receiver detected)

Using the PHY Reconfiguration Interface

Refer to the section PHY Reconfiguration Interface for details on how to use this
interface.

Follow the steps below to access registers in Table 110 on page 160 using the PHY
reconfiguration interface.
1. Enable the PHY reconfiguration interface using the IP Parameter Editor.
2. Set the Quad and address offset from which you want to read the status by
performing a read operation from the address xcvr_reconfig_addr_i[25:0]
using the PHY reconfiguration interface signals.
• xcvr_reconfig_read_i = 1’b1
• xcvr_reconfig_addr_i[25:0] = {5-bit Quad mapping, 21-bit address
offset}. Refer to Table 110 on page 160 for the address offset and bit
mapping.
• xcvr_reconfig_readdata_o[7:0] = Refer to Table 110 on page 160 for
the address offset and bit mapping.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

159
 7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

Table 110. PHY Reconfiguration Interface Register Map for PHY Status
PHY Offset Bit Position Register

0x000006 [7] PLLA state output status signal.

1'b1 indicates that PLLA is locked.

0x00000a [7] PLLB state output status signal.

1'b1 indicates that PLLB is locked.

Example 2: To read the PLLA status using the registers

1. Enable the PHY reconfiguration interface using the IP Parameter Editor.
2. Perform read from address 0x000006 to read the PLLA status output of Quad0
using the PHY reconfiguration interface signals.
• xcvr_reconfig_read_i = 1'b1
• xcvr_reconfig_addr_i[25:0] = 0x000006
• xcvr_reconfig_readdata_o[7:0] = 8'h80
• xcvr_reconfig_readdata_i[1] = 1'b1 (PLLA state output high indicating
PLL lock)

7.1.2. Debugging Data Transfer and Performance Issues

There are many possible reasons causing the PCIe link to stop transmitting data. The
PCI Express base specification defines three types of errors, outlined in the table
below:

Table 111. Error Types Defined by the PCI Express Base Specification
Type Responsible Agent Description

Correctable Hardware While correctable errors may affect

system performance, data integrity is
maintained.

Uncorrectable, non-fatal Device software Uncorrectable, non-fatal errors are

defined as errors in which data is lost,
but system integrity is maintained. For
example, the fabric may lose a
particular TLP, but it still works without
problems.

Uncorrectable, fatal System software Errors generated by a loss of data and

system failure are considered
uncorrectable and fatal. Software must
determine how to handle such errors:
whether to reset the link or implement
other means to minimize the problem.

Table 112. Correctable Error Status Register (AER)

Observation Issue Resolution

Receiver error bit set Physical layer error which may be due Use the configuration output interface,
to a PCS error when a lane is in L0, or or the Hard IP reconfiguration interface
a Control symbol being received in the and the flow chart in Figure 65 on page
wrong lane, or signal Integrity issues 155 to obtain more information about
where the link may transition from L0 the error.
to the Recovery state.
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

160
7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

Observation Issue Resolution

Bad DLLP bit set Data link layer error which may occur Use the configuration output interface
when a CRC verification fails. or the Hard IP reconfiguration interface
to obtain more information about the
error.

Bad TLP bit set Data link layer error which may occur Use the configuration output interface
when an LCRC verification fails or when or the Hard IP reconfiguration interface
a sequence number error occurs. to obtain more information about the
error.

Replay_num_rollover bit set Data link layer error which may be due Use the configuration output interface
to TLPs sent without success (no ACK) or the Hard IP reconfiguration interface
four times in a row. to obtain more information about the
error.

replay timer timeout status bit set Data link layer error which may occur Use the configuration output interface
when no ACK or NAK was received or the Hard IP reconfiguration interface
within the timeout period for the TLPs to obtain more information about the
transmitted. error.

Advisory non-fatal Transaction layer error which may be

due to higher priority uncorrectable
error detected.

Corrected internal error bits set Transaction layer error which may be Use the error interface, configuration
due to an ECC error in the internal output interface, or the Hard IP
Hard IP RAM. reconfiguration interface and DBI
registers to obtain more information
about the error.

Table 113. Uncorrectable Error Status Register (AER)

Observation Issue Resolution

Data link protocol error Data link layer error which may be due Use the configuration output interface,
to transmitter receiving an ACK/NAK Hard IP reconfiguration interface to
whose Seq ID does not correspond to obtain more information about the
an unacknowledged TLP or ACK error.
sequence number.

Surprise down error Data link layer error which may be due Use the error interface, configuration
to link_up_o getting deasserted output interface, Hard IP
during L0, indicating the physical layer reconfiguration interface and DBI
link is going down unexpectedly. registers to obtain more information
about the error.

Flow control protocol error Transaction layer error which can be Use the TX/RX flow control interface,
due to the receiver reporting more configuration output interface, Hard IP
than the allowed credit limit. reconfiguration interface to obtain
This error occurs when a component more information about the error.
does not receive updated flow control
credits with the 200 μs limit.

Poisoned TLP received Transaction layer error which can be Use the error interface, configuration
due to a received TLP with the EP bit output interface, configuration
set. intercept interface, Hard IP
reconfiguration interface to obtain
more information on the error and
determine the appropriate action.

Completion timeout Transaction layer error which can be Use the error interface, completion
due to a completion not received within timeout interface, configuration output
the required amount of time after a interface, Hard IP reconfiguration
non-posted request was sent. interface to obtain more information on
the error.
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

161
 7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

Observation Issue Resolution

Completer abort Transaction layer error which can be Use the configuration output interface,
due to a completer being unable to error interface, Hard IP reconfiguration
fulfill a request due to a problem with interface to obtain more information on
the requester or a failure of the the error.
completer.

Unexpected completion Transaction layer error which can be Use the configuration output interface,
due to a requester receiving a error interface, Hard IP reconfiguration
completion that doesn’t match any interface to obtain more information on
request awaiting a completion. the error.
The TLP is deleted by the Hard IP and
not presented to the Application Layer.

Receiver overflow Transaction layer error which can be Use the TX/RX flow control interface,
due to a receiver receiving more TLPs error interface, configuration output
than the available receive buffer space. interface, Hard IP reconfiguration
The TLP is deleted by the Hard IP and interface to obtain more information on
not presented to the Application Layer. the error.

Malformed TLP Transaction layer error which can be Use the error interface, configuration
due to errors in the received TLP output interface, Hard IP
header. reconfiguration interface to obtain
The TLP is deleted by the Hard IP and more information on the error.
not presented to the Application Layer.

ECRC error Transaction layer error which can be Use the configuration output interface,
due to an ECRC check failure at the Hard IP reconfiguration interface to
receiver despite the fact that the TLP is obtain more information on the error.
not malformed and the LCRC check is
valid.
The Hard IP block handles this TLP
automatically. If the TLP is a non-
posted request, the Hard IP block
generates a completion with a
completer abort status. The TLP is
deleted by the Hard IP and not
presented to the Application Layer.

Unsupported request Transaction layer error which can be Use the configuration output interface,
due to the completer being unable to error interface, Hard IP reconfiguration
fulfill the request. interface to obtain more information on
The TLP is deleted in the Hard IP block the error.
and not presented to the Application
Layer. If the TLP is a non-posted
request, the Hard IP block generates a
completion with Unsupported Request
status.

ACS violation Transaction layer error which can be Use the configuration output interface,
due to access control error in the error interface, Hard IP reconfiguration
received posted or non-posted request. interface to obtain more information on
the error.

Uncorrectable internal error Transaction layer error which can be Use the error interface, configuration
due to an internal error that cannot be output interface, Hard IP
corrected by the hardware. reconfiguration interface and DBI
registers to obtain more information on
the error.
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

162
7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

Observation Issue Resolution

Atomic egress blocked Use the error interface, configuration

output interface, Hard IP
reconfiguration interface to obtain
more information on the error.

TLP prefix blocked EP or RP only Use the error interface, configuration

output interface, Hard IP
reconfiguration interface to obtain
more information on the error.

Poisoned TLP egress blocked EP or RP only Use the error interface, configuration
output interface, configuration
intercept interface, Hard IP
reconfiguration interface to obtain
more information on the error.

Use the debug tools mentioned in the next two sections for debugging link training
issues observed on the PCI Express link when using the P-Tile Avalon-ST IP for PCI
Express.

Related Information
PCI Express Base Specification Revision 4.0 version 1.0

7.1.2.1. Advanced Error Reporting (AER)

Each PCI Express compliant device must implement a basic level of error management
and can optionally implement advanced error management. The PCI Express
Advanced Error Reporting Capability is an optional Extended Capability that may be
implemented by PCI Express device functions supporting advanced error control and
reporting.

The P-Tile Avalon-ST IP for PCI Express implements both basic and advanced error
reporting. Error handling for a Root Port is more complex than that of an Endpoint. In
this IP, the Physical Functions (PFs) are always capable of AER (enabled by default).
There is no AER implementation for Virtual Functions (VFs).

Use the AER capability of the IP to identify the type of error and the protocol stack
layer in which the error may have occurred. Refer to the PCI Express Capability
Structures section of the Configuration Space Registers appendix for the AER
Extended Capability Structure and the associated registers.

7.1.2.2. Second-Level Debug Tools

Use the following debug tools for second-level debug of any issue observed on the PCI
Express link when using P-Tile:

Using the Configuration Output Interface

• Refer to the section Configuration Output Interface on page 87 for details on this
interface and the address map.

Using the Error Interface

• Refer to the section Error Interface on page 79 for details on this interface and the
address map.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

163
 7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

Using the Configuration Intercept Interface

• Refer to the section Configuration Intercept Interface (EP Only) on page 91 for
details on this interface and the address map.

Using the TX/RX Flow Control Interfaces

• Refer to the sections TX Flow Control Interface on page 68 and RX Flow Control
Interface on page 59 for details on these interfaces and their address maps.

Using the Hard IP Reconfiguration Interface

• Refer to the section Hard IP Reconfiguration Interface on page 93 for details on
this interface and the address map.

Using the PHY Reconfiguration Interface

• Refer to the section PHY Reconfiguration Interface on page 101 for details on this
interface and the address map.

7.2. Debug Toolkit

7.2.1. Overview
The P-Tile Debug Toolkit is a System Console-based tool for P-Tile that provides real-
time control, monitoring and debugging of the PCIe links at the Physical Layer.

The P-Tile Debug Toolkit allows you to:

• View protocol and link status of the PCIe links.
• View PLL and per-channel status of the PCIe links.
• Control the channel analog settings.
• View the receiver eye and measure the eye height and width.
• Indicate the presence of a re-timer connected between the link partners.

The following figure provides an overview of the P-Tile Debug Toolkit in the P-Tile
Avalon-ST IP for PCI Express.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

164
7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

Figure 68. Overview of the P-Tile Debug Toolkit

intel_pcie_ptile_ast

P-Tile Debug Toolkit

PCle Config Space Registers

AVMM (Port 0)
..
.
hip-reconfig_* PCle Config Space Registers
AVMM (Port 3)
NPDME

AVMM PHY Registers (Quad 0)

..
System
.
xcvr_reconfig_*
Console
GUI AVMM PHY Registers (Quad 3)

When you enable the P-Tile Debug Toolkit, the intel_pcie_ptile_ast module of
the generated IP includes the Debug Toolkit modules and related logic as shown in the
figure above.

Drive the Debug Toolkit from a System Console. The System Console connects to the
Debug Toolkit via an Native PHY Debug Master Endpoint (NPDME). Make this
connection via an Intel FPGA Download Cable.

The PHY reconfiguration interface clock (xcvr_reconfig_clk) is used to clock the

following interfaces:
• The NPDME module
• PHY reconfiguration interface (xcvr_reconfig)
• Hard IP reconfiguration interface (hip_reconfig)

Provide a clock source (50 MHz - 125 MHz, 100 MHz recommended clock frequency) to
drive the xcvr_reconfig_clk clock. Use the output of the Reset Release Intel FPGA
IP to drive the ninit_done, which provides the reset signal to the NPDME module.

Note: When you enable the P-Tile Debug Toolkit, the Hard IP Reconfiguration interface is
enabled by default.

When you run a dynamically-generated design example on the Intel Development Kit,
make sure that clock and reset signals are connected to their respective sources and
appropriate pin assignments are made. Here are some sample .qsf assignments for
the Debug Toolkit:
• set_location_assignment PIN_A31 -to p0_hip_reconfig_clk_clk
• set_location_assignment PIN_C23 -to xcvr_reconfig_clk_clk

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

165
 7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

7.2.2. Enabling the P-Tile Debug Toolkit

To enable the P-Tile Debug Toolkit in your design, enable the option Enable Debug
Toolkit in the PCIe Configuration, Debug and Extension options tab of the Intel
FPGA P-Tile Avalon-ST IP for PCI Express.

Note: When you enable the P-Tile Debug Toolkit in the IP, the Hard IP reconfiguration
interface and the PHY reconfiguration interface will be used by the Debug Toolkit.
Hence, you will not be able to drive logic on these interfaces from the FPGA fabric.

7.2.3. Launching the P-Tile Debug Toolkit

Use the design example you compiled by following the Quick Start Guide to familiarize
yourself with the P-Tile Debug Toolkit. Follow the steps in the Generating the Design
Example and Compiling the Design Example to generate the SRAM Object File, (.sof)
for this design example.

To use the P-Tile Debug Toolkit, download the .sof to the Intel Development Kit. Then,
open the System Console and load the design to the System Console as well. Loading
the .sof to the System Console allows the System Console to communicate with the
design using NPDME. NPDME is a JTAG-based Avalon-MM master. It drives Avalon-MM
slave interfaces in the PCIe design. When using NPDME, the Intel Quartus Prime
software inserts the debug interconnect fabric to connect with JTAG.

Here are the steps to complete these tasks:

1. Use the Intel Quartus Prime Programmer to download the .sof to the Intel FPGA
Development Kit.
Note: To ensure correct operation, use the same version of the Intel Quartus
Prime Programmer and Intel Quartus Prime Pro Edition software that you
used to generate the .sof.
2. To load the design into System Console:
a. Launch the Intel Quartus Prime Pro Edition software.
b. Start System Console by choosing Tools, then System Debugging Tools,
then System Console.
c. On the System Console File menu, select Load design and browse to the .sof
file.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

166
7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

d. Select the .sof and click OK. The .sof loads to the System Console.
3. The System Console Toolkit Explorer window will list all the DUTs in the design
that have the P-Tile Debug Toolkit enabled.
a. Select the DUT with the P-Tile Debug Toolkit you want to view. This will open
the Debug Toolkit instance of that DUT in the Details window.

b. Click on the ptile_debug_toolkit_avst to open that instance of the Toolkit.

Once the Debug Toolkit is initialized and loaded, you will see the following
message in the Messages window: “Initializing P-Tile debug toolkit –
done”.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

167
 7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

c. A new window Main view will open with a view of all the channels in that
instance.

7.2.4. Using the P-Tile Debug Toolkit

The following sections describe the different tabs and features available in the Debug
Toolkit.

A. Main View

The main view tab lists a summary of the transmitter and receiver settings per
channel for the given instance of the PCIe IP.

The following table shows the channel mapping when using bifurcated ports.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

168
7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

Table 114. Channel Mapping for Bifurcated Ports

Toolkit Channel X16 Mode 2X8 Mode 4x4 Mode

Lane 0 Lane 0 Lane 0 Lane 0

Lane 1 Lane 1 Lane 1 Lane 1

Lane 2 Lane 2 Lane 2 Lane 2

Lane 3 Lane 3 Lane 3 Lane 3

Lane 4 Lane 4 Lane 4 Lane 0

Lane 5 Lane 5 Lane 5 Lane 1

Lane 6 Lane 6 Lane 6 Lane 2

Lane 7 Lane 7 Lane 7 Lane 3

Lane 8 Lane 8 Lane 0 Lane 0

Lane 9 Lane 9 Lane 1 Lane 1

Lane 10 Lane 10 Lane 2 Lane 2

Lane 11 Lane 11 Lane 3 Lane 3

Lane 12 Lane 12 Lane 4 Lane 0

Lane 13 Lane 13 Lane 5 Lane 1

Lane 14 Lane 14 Lane 6 Lane 2

Lane 15 Lane 15 Lane 7 Lane 3

B. Toolkit Parameters

The Toolkit parameters window has 2 sub-tabs.

B.1. P-Tile Information

This lists a summary of the P-Tile PCIe IP parameter settings in the PCIe IP Parameter
Editor when the IP was generated, as read by the P-Tile Debug Toolkit when initialized.

All the information is read-only.

Use the Get P-tile Info button to read the settings.

Table 115. P-Tile Available Parameter Settings

Parameter Values Descriptions

Indicates the Vendor ID as set in the IP

Intel Vendor ID 1172
Parameter Editor.

Protocol PCIe Indicates the Protocol.

HIP Type Root Port, End Point Indicates the Hard IP Port type.

intel_pcie_ptile_ast,
Intel IP Type Indicates the IP type used.
intel_pcie_ptile_avmm

Indicates the advertised speed as

Advertised speed Gen3, Gen4
configured in the IP Parameter Editor.

Indicates the advertised width as

Advertised width x16, x8, x4
configured in the IP Parameter Editor.
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

169
 7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

Parameter Values Descriptions

Indicates the negotiated speed during

Negotiated speed Gen3, Gen4
link training.

Indicates the negotiated link width

Negotiated width x16, x8, x4
during link training.

Link status Link up, link down Indicates if the link (DL) is up or not.

Indicates if a retimer was detected

Retimer 1 Detected, not detected between the Root Port and the
Endpoint.

Indicates if a retimer was detected

Retimer 2 Detected, not detected between the Root Port and the
Endpoint.

Figure 69. Example of P-Tile Parameter Settings

B.2. PCIe Configuration Space

This lists a summary of the P-Tile PCIe configuration settings of the PCIe configuration
space registers, as read by the P-Tile Debug Toolkit when initialized.

All the information is read-only.

Use the Read cfg space button to read the settings.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

170
7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

Figure 70. Example of P-Tile PCIe Configuration Settings

C. Channel Parameters

The channel parameters window allows you to monitor and control the transmitter and
receiver settings for a given channel. It has the following 2 sub-windows.

C.1. TX Path

This tab allows you to monitor and control the transmitter settings for the channel
selected. Use the TX Refresh button to read the settings, TX Apply Ch to apply the
settings to the selected channel, and TX apply all to apply the settings to all
channels.

Table 116. Transmitter Settings

Parameters Values Descriptions

Indicates reference clock is

enabled for the PHY.
Enable: Reference clock is
Refclk enable Enable, Disable
enabled for the PHY.
Disable: Reference clock is
PHY Status disabled for the PHY.

Indicates the PHY is in reset

mode.
PHY reset Normal, Reset
Normal: PHY is out of reset.
Reset: PHY is in reset.

Indicates if TX lane is
enabled in the PHY.
TX Status TX Lane enable Enable, Disable
Enable: TX lane is enabled in
the PHY.
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

171
 7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

Parameters Values Descriptions

Disable: TX lane is disabled

in the PHY.

Indicates if TX driver is
enabled and serial data is
transmitted.
Enable: TX driver for the
TX Data enable Enable, Disable corresponding lane is
enabled.
Disable: TX driver for the
corresponding lane is
disabled.

Indicates if TX (TX datapath,

TX settings) is in reset or
normal operating mode.
TX Reset Normal, Reset
Normal: TX is in normal
operating mode.
Reset: TX is in reset.

Indicates if the TX PLL is

powered on or powered
down. This is dependent on
the PLL selected as indicated
by TX PLL select.
There is one set of PLLs per
Quad. The TX path of each
channel reads out the PLL
status corresponding to that
Quad.
• TX path for Ch0 to 3:
TX PLL enable Enable, Disable Status of PLLs in Quad0
• TX path for Ch4 to 7:
Status of PLLs in Quad1
• TX path for Ch8 to 11:
Status of PLLs in Quad2
• TX path for Ch12 to 15:
Status of PLLs in Quad3
Enable: TX PLL is powered
on.
Disable: TX PLL is powered
TX PLL down.

Indicates which PLL is

selected.
There is one set of PLLs per
Quad. The TX path of each
channel reads out the PLL
status corresponding to that
Quad.
PLLA: Gen1/Gen2
TX PLL select • TX path for Ch0 to 3:
PLLB: Gen3/Gen4 Status of PLLs in Quad0
• TX path for Ch4 to 7:
Status of PLLs in Quad1
• TX path for Ch8 to 11:
Status of PLLs in Quad2
• TX path for Ch12 to 15:
Status of PLLs in Quad3

Indicates if TX PLL is locked.

This is dependent on the PLL
TX PLL lock Green, Red
selected as indicated by TX
PLL select.
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

172
7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

Parameters Values Descriptions

There is one set of PLLs per

Quad. The TX path of each
channel reads out the PLL
status corresponding to that
Quad.
• TX path for Ch0 to 3:
Status of PLLs in Quad0
• TX path for Ch4 to 7:
Status of PLLs in Quad1
• TX path for Ch8 to 11:
Status of PLLs in Quad2
• TX path for Ch12 to 15:
Status of PLLs in Quad3
Green: TX PLL is locked.
Red: TX PLL is not locked.

Indicates the transmitter

Gen3: 15 current boost level when the
Iboost level
Gen4: 15 TX amplitude boost mode is
enabled.

Indicates if the TX swing

boost level is enabled.
TX VOD Gen3: Enable Enable: TX swing boost is
Vboost en
Gen4: Enable enabled.
Disable: TX swing boost is
disabled.

Gen3: 5 Indicates the TX Vboost

Vboost level
Gen4: 5 level.

Gen3: 20 (Preset 8) Indicates transmitter driver

Pre-shoot coefficient output pre-emphasis (pre-
Gen4: 0 (Preset 0) shoot coefficient).

Gen3: 30 (Preset 8) Indicates transmitter driver

TX Equalization Main coefficient output pre-emphasis (main
Gen4: 30 (Preset 0) coefficient).

Gen3: 20 (Preset 8) Indicates transmitter driver

Post coefficient output pre-emphasis (post
Gen4: 40 (Preset 0) coefficient).

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

173
 7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

Figure 71. Example of Transmitter Settings

C.2. RX Path

This tab allows you to monitor and control the receiver settings for the channel
selected. Use the RX Refresh button to read the settings, RX Apply Ch to apply the
settings to the selected channel, and RX apply all to apply the settings to all
channels.

Table 117. Receiver Settings

Parameters Values Descriptions

Indicates if RX lane is
enabled in the PHY.
Enable: RX lane is enabled
RX Lane enable Enable, Disable
in the PHY.
Disable: RX lane is disabled
in the PHY.

Indicates if RX driver is
enabled and serial data is
transmitted.
Enable: RX driver for the
RX Data enable Enable, Disable corresponding lane is
enabled.
Disable: RX driver for the
RX Status corresponding lane is
disabled.

Indicates if RX (RX datapath,

RX settings) is in reset or
normal operating mode.
RX Reset Normal, Reset
Normal: RX is in normal
operating mode.
Reset: RX is in reset.

Indicates if the receiver has

lost the signal.
RX LOS <1,0> 1: Receiver loss of signal.
0: Receiver has a data
signal.
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

174
7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

Parameters Values Descriptions

Indicates the CDR lock state.

CDR Lock Green, Red Green: CDR is locked.
Red: CDR is not locked.

RX CDR Indicates the CDR lock

mode.
Locked to Reference (LTR),
CDR Mode LTR: CDR is locked to
Locked to Data (LTD)
reference clock.
LTD: CDR is locked to data.

Gen3: Gen3 adaptation

mode. Indicates the RX adaptation
Adapt Mode
Gen4: Gen4 adaptation mode.
mode.

Indicates if the receiver is in

continuous adaptation.
Gen3: 1 • 0 - continuous adaptation
Adapt Continuous
Gen4: 1 off.
• 1 - continuous adaptation
on.

Gen3: 0 Indicates the RX equalization

RX ATT
Gen4: 0 attenuation level.

Gen3: 12 Indicates the RX CTLE boost

RX CTLE Boost
Gen4: 16 value.

Gen3: 2 Indicates the RX CTLE pole

RX CTLE Pole
Gen4: 2 value.

Gen3: 5 Indicates the RX AFE first

RX VGA1
Gen4: 5 stage VGA gain value.

Gen3: 5 Indicates the RX AFE second

RX VGA2
Gen4: 5 stage VGA gain value.
RX Equalization
Indicates the Receiver Figure
of Merit (FOM) / quality of
the received data eye. A
higher value indicates better
RX FOM <0 - 255> link equalization, with 8'd0
indicating the worst
equalization setting and
8'd255 indicating the best
equalization setting.

Indicates DFE adaptation is

enabled for taps 1 - 5.
Enable: DFE adaptation is
DFE Enable Enable, Disable
enabled for taps 1 - 5.
Disable: DFE adaptation is
disabled for taps 1 - 5.

Indicates the adapted value

of DFE tap 1. This is a
DFE Tap1 adapted value <-128 to 127>
signed input (two's
complement encoded).

Indicates the adapted value

of DFE tap 2. This is a
DFE Tap2 adapted value <-32 to 31>
signed input (two's
complement encoded).
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

175
 7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

Parameters Values Descriptions

Indicates the adapted value

of DFE tap 3. This is a
DFE Tap3 adapted value <-32 to 31>
signed input (two's
complement encoded).

Indicates the adapted value

of DFE tap 4. This is a
DFE Tap4 adapted value <-32 to 31>
signed input (two's
complement encoded).

Indicates the adapted value

of DFE tap 5. This is a
DFE Tap5 adapted value <-32 to 31>
signed input (two's
complement encoded).

Figure 72. Example of Receiver Settings

D. Eye Viewer

The P-Tile Debug Toolkit supports running eye tests for Intel devices with P-Tile. The
Eye Viewer tool allows you to set up and run eye tests, monitoring bit errors.
1. In the System Console Tools menu option, click on Eye View Tool.

Figure 73. Opening the Eye Viewer

2. This will open a new tab Eye View Tool next to the Main View tab. Choose the
instance and channel for which you want to run the eye view tests.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

176
7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

Figure 74. Opening the Instance and Channel

3. Choose the eye vertical step setting from the drop-down menu. The eye view tool
allows you to choose between vertical step sizes of 1, 2, 4, 8.
Note: The time taken for the eye view tool to draw the eye varies with different
vertical step sizes (8 results in a faster eye plot when compared to 1).

Figure 75. Choosing the Step Size

4. The messages window displays information messages to indicate the eye view
tool's progress.

Figure 76. Eye View Tool Messages

5. Once the eye plot is complete, the eye height, eye width and eye diagram are
displayed.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

177
 7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

Figure 77. Sample Eye Plot

7.2.5. Enabling the P-Tile Link Inspector

To enable the Link Inspector, enable the option Enable Debug Toolkit in the PCIe
Configuration, Debug and Extension Options tab. The PCIe Link Inspector is
enabled by default when the Enable Debug Toolkit is enabled.

Figure 78. Enabling the Link Inspector

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

178
7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

7.2.6. Using the P-Tile Link Inspector

The Link Inspector is found under the PCIe Link Inspector tab after opening the
Debug Toolkit:

Figure 79. View of the Link Inspector

When the Dump LTSSM Sequence to Text File button is initially clicked, a text file
(ltssm_sequence_dump_p*.txt) with the LTSSM information is created in the
location from where the System Console window is opened. Depending on the PCIe
topology, there can be up to four text files. Subsequent LTSSM sequence dumps will
append to the respective files.

Note: If you open System Console in a directory that is not writable, the text file will not be
generated. To avoid this issue, open System Console from the Command Prompt
window (on a Windows system) or change the directory's permission settings to
writable.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

179
 7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

Figure 80. Example LTSSM Sequence Dump (Beginning)

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

180
7. Troubleshooting/Debugging
UG-20225 | 2021.02.18

Figure 81. Example LTSSM Sequence Dump (End)

Each LTSSM monitor has a FIFO storing the time values and captured LTSSM states.
When you choose to dump out the LTSSM states, reads are dependent on the FIFO
elements and will empty out the FIFO.

The Link Inspector only writes to its FIFO if there is a state transition. In cases where
the link is stable in L0, there will be no write and hence no text file will be dumped.

When you want to dump the LTSSM sequence, a single read of the FIFO status of the
respective core is performed. Depending on the empty status and how many entries
are in the FIFO, successive reads are executed.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

181
 UG-20225 | 2021.02.18

Send Feedback

8. Document Revision History

8.1. Document Revision History for the Intel FPGA P-Tile Avalon
Streaming IP for PCI Express User Guide
Intel Quartus IP
Document Version Changes
Prime Version Version

Replaced pin_perst_n with p<n>_pin_perst_n in the Table

2021.02.18 20.4 4.0.0
in section SignalTapII Logic Analyzer.

Updated the Hard IP Mode options in the Top-Level Settings

2021.01.19 20.4 4.0.0
section.

Added Function-Level Reset (FLR) for PF and VF timing diagrams

2020.12.18 20.4 4.0.0
to the Function-Level Reset (FLR) section.

Added parameters to enable the independent resets for the x8x8

bifurcated mode to the Parameters chapter.
Added a note specifying the default value of the Extended Tag
field in the Device Capabilities register to the table in the SR-IOV
Supported Features List section.
Updated the description of the
p0_flr_completed_pf_i[7:0] signals in the Function-Level
2020.12.14 20.4 4.0.0
Reset (FLR) Interface (EP Only) section.
Updated the table Port Mode Options in TLP Bypass in the Top-
Level Settings section to clarify the combinations in which TLP
Bypass can be enabled or disabled when multiple ports are
available.
Updated the directions of signals in the Avalon-ST TX Interface
section to match the signal directions in the block symbol.

Added descriptions for the Device Serial Number tab, the ACS
tab to the Parameters chapter.
2020.10.05 20.3 3.1.0
Updated the description of the p0_flr_rcvd_pf_o[7:0] signal
bus in the Function-Level Reset (FLR) Interface section.

Removed configurations that require the Adapter (Gen4 x8 512-

bit and Gen4 x4 256-bit) from the Top-Level Settings section
because they are not supported in the 20.2 release of Intel
Quartus Prime.
2020.07.10 20.2 3.0.0
Added description for the Link Inspector in the Debug Toolkit
chapter.
Added support for the Modelsim simulator to the Features
section.

Added the lane reversal and polarity inversion support to the

Features section.
2020.06.22 20.2 3.0.0
Updated the bit ranges for the Next Capability Offset and Version
fields in the Intel-Defined VSEC Capability Registers section.
continued...

Intel Corporation. All rights reserved. Agilex, Altera, Arria, Cyclone, Enpirion, Intel, the Intel logo, MAX, Nios,
Quartus and Stratix words and logos are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or
other countries. Intel warrants performance of its FPGA and semiconductor products to current specifications in ISO
accordance with Intel's standard warranty, but reserves the right to make changes to any products and services 9001:2015
at any time without notice. Intel assumes no responsibility or liability arising out of the application or use of any Registered
information, product, or service described herein except as expressly agreed to in writing by Intel. Intel
customers are advised to obtain the latest version of device specifications before relying on any published
information and before placing orders for products or services.
*Other names and brands may be claimed as the property of others.
8. Document Revision History
UG-20225 | 2021.02.18

Intel Quartus IP
Document Version Changes
Prime Version Version

Removed the Enable independent pin_perst parameter from

the Avalon Parameters section and the reset pin_perst_2_n
from the Interface Reset Signals section. The independent
pin_perst option is not supported in the 20.2 release of Intel
Quartus Prime.

Added clarification that VCS is the only simulator supported in

the 20.1 release of Intel Quartus Prime. Also added a note
2020.04.30 20.1 2.0.0
stating that PIPE mode simulations are not supported in this
release.

Added notes to the Avalon-ST RX Interface and Avalon-ST TX

Interface stating that the segmented (split) Avalon-ST bus
2020.04.29 20.1 2.0.0
interface needs to be leveraged to achieve the expected Gen4
x16 performance.

Updated the title of the document to Intel FPGA P-Tile Avalon

streaming IP for PCI Express User Guide to meet new legal
naming guidelines.
Added new parameters Enable Rx Buffer Limit Ports, P-tile
Sim Mode and Enable independent pin_perst.
2020.04.22 20.1 2.0.0
Added diagrams showing examples of buffer limits updates to
the RX Flow Control Interface and TX Flow Control Interface
sections.
Removed Notes stating that data parity is not supported because
that feature is available in Intel Quartus Prime 20.1.

Added information about the availability of the CvP Init and CvP
Update features in Intel Stratix 10 DX and Intel Agilex devices to
the Features section.
2020.01.16 19.4 1.1.0 Added the rx_st_tlp_abort_o[1:0] signals to the Avalon-ST
RX Interface section.
Removed the app_ready_entr_l23_i signal from the Power
Management Interface section.

Added parameters in Intel Quartus Prime to control PASID and

2019.12.16 19.4 1.1.0 LTR.
Added MSI extended data support.

Added resource utilization numbers for the PIO design example

in Intel Stratix 10 DX devices.
2019.11.04 19.3 1.0.0
Added a step to select Intel Stratix 10 DX devices in the
Generating the Design Example section.

Added the description and usage instructions for the P-Tile

Debug Toolkit.
2019.10.23 19.3 1.0.0
Added an Appendix chapter on how to use the Avery BFM to run
Gen4 x16 simulations.

2019.07.19 19.2 1.0.0 Added features such as SR-IOV support and VirtIO support.

2019.05.03 19.1.1 Initial release.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

183
 UG-20225 | 2021.02.18

Send Feedback

A. Configuration Space Registers

A.1. Configuration Space Registers

In addition to accessing the Endpoint's configuration space registers by sending
Configuration Read/Write TLPs via the Avalon-ST interface, the application logic can
also gain read access to these registers via the Configuration Output Interface
(tl_cfg*). Furthermore, the Hard IP Reconfiguration Interface (a User Avalon-MM
interface) also provides read/write access to these registers.

For signal timings on the User Avalon-MM interface, refer to the Avalon Interface
Specifications document.

The table PCIe Configuration Space Registers describes the registers for each PF. To
calculate the address for a particular register in a particular PF, add the offset for that
PF from the table Configuration Space Offsets to the byte address for that register as
given in the table PCIe Configuration Space Registers.

Table 118. Configuration Space Offsets

Registers User Avalon-MM Offsets

Physical function 0 0x00000

Physical function 1 0x10000

Physical function 2 0x20000

Physical function 3 0x30000

Physical function 4 0x40000

Physical function 5 0x50000

Physical function 6 0x60000

Physical function 7 0x70000

Port Configuration and Status Register 0x104000

Debug (DBI) Register 0x104200, 0x104204

Table 119. PCIe Configuration Space Registers for x16/x8/x4 Controllers

Hard IP Configuration Space Corresponding Section in PCIe
Byte Address
Register Specification

x16 (Port 0) = 0x000 : 0x03C PCI Header Type 0/1 Configuration Type 0/1 Configuration Space Header
x8 (Port 1) = 0x000 : 0x03C Registers
x4 (Ports 2,3) = 0x000 : 0x03C

x16 (Port 0) = 0x040 : 0x044 Power Management PCI Power Management Capability
x8 (Port 1) = 0x040 : 0x044 Structure
x4 (Ports 2,3) = 0x040 : 0x044
continued...

Intel Corporation. All rights reserved. Agilex, Altera, Arria, Cyclone, Enpirion, Intel, the Intel logo, MAX, Nios,
Quartus and Stratix words and logos are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or
other countries. Intel warrants performance of its FPGA and semiconductor products to current specifications in ISO
accordance with Intel's standard warranty, but reserves the right to make changes to any products and services 9001:2015
at any time without notice. Intel assumes no responsibility or liability arising out of the application or use of any Registered
information, product, or service described herein except as expressly agreed to in writing by Intel. Intel
customers are advised to obtain the latest version of device specifications before relying on any published
information and before placing orders for products or services.
*Other names and brands may be claimed as the property of others.
A. Configuration Space Registers
UG-20225 | 2021.02.18

Hard IP Configuration Space Corresponding Section in PCIe

Byte Address
Register Specification

x16 (Port 0) = 0x050 : 0x064 MSI Capability MSI Capability Structure, see also PCI
x8 (Port 1) = 0x050 : 0x064 Local Bus Specification
x4 (Ports 2,3) = 0x050 : 0x064

x16 (Port 0) = 0x070 : 0x0A8 PCI Express Capability PCI Express Capability Structure
x8 (Port 1) = 0x070 : 0x0A8
x4 (Ports 2,3) = 0x070 : 0x0A8

x16 (Port 0) = 0x0B0 : 0x0B9 MSI-X Capability MSI-X Capability Structure, see also
x8 (Port 1) = 0x0B0 : 0x0B9 PCI Local Bus Specification
x4 (Ports 2,3) = 0x0B0 : 0x0B9

x16 (Port 0) = 0x0BC : 0x0FC Reserved N/A

x8 (Port 1) = 0x0BC : 0x0FC
x4 (Ports 2,3) = 0x0BC : 0x0FC

x16 (Port 0) = 0x100 : 0x144 Advanced Error Reporting (AER) Advanced Error Reporting Capability
x8 (Port 1) = 0x100 : 0x144 Structure
x4 (Ports 2,3) = 0x100 : 0x144

x16 (Port 0) = 0x148 : 0x164 Virtual Channel Capability Virtual Channel Capability Structure
x8 (Port 1) = 0x148 : 0x164
x4 (Ports 2,3) = 0x148 : 0x164

x16 (Port 0) = 0x178 : 0x17C Alternative Routing-ID Implementation ARI Capability Structure
x8 (Port 1) = 0x178 : 0x17C (ARI)
x4 (Ports 2,3) = N/A

x16 (Port 0) = 0x188 : 0x1B4 Secondary PCI Express Extended PCI Express Extended Capability
x8 (Port 1) = 0x188 : 0x1A4 Capability Header
x4 (Ports 2,3) = 0x188 : 0x1A4

x16 (Port 0) = 0x1B8 : 0x1E4 Physical Layer 16.0 GT/s Extended Physical Layer 16.0 GT/s Extended
x8 (Port 1) = 0x1A8 : 0x1CC Capability Capability Structure
x4 (Ports 2,3) = 0x1A8 : 0x1C8

x16 (Port 0) = 0x1E8 : 0x22C Margining Extended Capability Margining Extended Capability
x8 (Port 1) = 0x1D0 : 0x1F4 Structure
x4 (Ports 2,3) = 0x1CC : 0x1E0

x16 (Port 0) = 0x230 : 0x26C SR-IOV Capability SR-IOV Capability Structure

x8 (Port 1) = 0x1F8 : 0x234
x4 (Ports 2,3) = N/A

x16 (Port 0) = 0x270 : 0x2F8 TLP Processing Hints (TPH) Capability TLP Processing Hints (TPH) Capability
x8 (Port 1) = 0x238 : 0x2C0 Structure
x4 (Ports 2,3) = 0x1E4 : 0x26C

x16 (Port 0) = 0x2FC : 0x300 Address Translation Services (ATS) Address Translation Services Extended
x8 (Port 1) = 0x2C4 : 0x2C8 Capability Capability (ATS) in Single Root I/O
Virtualization and Sharing Specification
x4 (Ports 2,3) = N/A

x16 (Port 0) = 0x30C : 0x314 Access Control Services (ACS) Access Control Services (ACS)
x8 (Port 1) = 0x2D4 : 0x2DC Capability Capability
x4 (Ports 2,3) = 0x280 : 0x288

x16 (Port 0) = 0x318 : 0x324 Page Request Services (PRS) Capability Page Request Services (PRS) Capability
x8 (Port 1) = 0x2E0 : 0x2EC
x4 (Ports 2,3) = N/A
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

185
 A. Configuration Space Registers
UG-20225 | 2021.02.18

Hard IP Configuration Space Corresponding Section in PCIe

Byte Address
Register Specification

x16 (Port 0) = 0x328 : 0x32C Latency Tolerance Reporting (LTR) Latency Tolerance Reporting (LTR)
x8 (Port 1) = 0x2F0 : 0x2F4 Capability Capability
x4 (Ports 2,3) = N/A

x16 (Port 0) = 0x330 : 0x334 Process Address Space (PASID) Process Address Space (PASID)
x8 (Port 1) = 0x2F8 : 0x2FC Capability Capability Structure
x4 (Ports 2,3) = N/A

x16 (Port 0) = 0x338 : 0x434 RAS D.E.S. Capability (VSEC)

x8 (Port 1) = 0x300 : 0x3FC
x4 (Ports 2,3) = 0x2AC : 0x3A8

x16 (Port 0) = 0x470 : 0x478 Data Link Feature Extended Capability

x8 (Port 1) = 0x438 : 0x440
x4 (Ports 2,3) = 0x3E4 : 0x3EC

x16 (Port 0) = 0xD00 : 0xD58 Intel-defined VSEC

x8 (Port 1) = 0xD00 : 0xD58
x4 (Ports 2,3) = 0xD00 : 0xD58

A.1.1. Register Access Definitions

This document uses the following abbreviations when describing register accesses.

Table 120. Register Access Abbreviations

Abbreviation Meaning

RW Read and write access

RO Read only

WO Write only

RW1C Read write 1 to clear

RW1CS Read write 1 to clear sticky

RWS Read write sticky

Note: Sticky bits are not initialized or modified by hot reset or function-level reset.

A.1.2. PCIe Configuration Header Registers

The Corresponding Section in PCIe Specification column in the tables in the
Configuration Space Registers section lists the appropriate sections of the PCI Express
Base Specification that describe these registers.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

186
A. Configuration Space Registers
UG-20225 | 2021.02.18

Figure 82. PCIe Type 0 Configuration Space Registers - Byte Address Offsets and Layout

31 24 23 16 15 87 0
0x000 Device ID Vendor ID
0x004 Status Command
0x008 Class Code Revision ID
0x00C 0x00 Header Type 0x00 Cache Line Size
0x010 BAR Registers
0x014 BAR Registers
0x018 BAR Registers
0x01C BAR Registers
0x020 BAR Registers
0x024 BAR Registers
0x028 Reserved
0x02C Subsystem Device ID Subsystem Vendor ID
0x030 Reserved
0x034 Reserved Capabilities Pointer
0x038 Reserved
0x03C 0x00 Interrupt Pin Interrupt Line
Figure 83. PCIe Type 1 Configuration Space Registers - Byte Address Offsets and Layout
31 24 23 16 15 87 0
0x0000 Device ID Vendor ID
0x004 Status Command
0x008 Class Code Revision ID
0x00C BIST Header Type Primary Latency Timer Cache Line Size
0x010 BAR Registers
0x014 BAR Registers
0x018 Secondary Latency Timer Subordinate Bus Number Secondary Bus Number Primary Bus Number
0x01C Secondary Status I/O Limit I/O Base
0x020 Memory Limit Memory Base
0x024 Prefetchable Memory Limit Prefetchable Memory Base
0x028 Prefetchable Base Upper 32 Bits
0x02C Prefetchable Limit Upper 32 Bits
0x030 I/O Limit Upper 16 Bits I/O Base Upper 16 Bits
0x034 Reserved Capabilities Pointer
0x038 Expansion ROM Base Address
0x03C Bridge Control Interrupt Pin Interrupt Line

Related Information
PCI Express Base Specification 4.0

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

187
 A. Configuration Space Registers
UG-20225 | 2021.02.18

A.1.3. PCI Express Capability Structures

The layouts of the most basic Capability Structures are provided below. Refer to the
PCI Express Base Specification for more information about these registers.

Figure 84. Power Management Capability Structure - Byte Address Offsets and Layout
31 24 23 16 15 87 0
0x040 Capabilities Register Next Cap Ptr Capability ID
PM Control/Status
0x04C Data Power Management Status and Control
Bridge Extensions

Figure 85. MSI Capability Structure

31 24 23 16 15 87 0
Message Control
0x050 Configuration MSI Control Status Next Cap Ptr Capability ID
Register Field Descriptions
0x054 Message Address
0x058 Message Upper Address
0x05C Reserved Message Data

Figure 86. PCI Express Capability Structure - Byte Address Offsets and Layout
In the following table showing the PCI Express Capability Structure, registers that are not applicable to a
device are reserved.
31 24 23 16 15 87 0
PCI Express
0x070 PCI Express Capabilities Register Next Cap Pointer
Capabilities ID
0x074 Device Capabilities
0x078 Device Status Device Control
0x07C Link Capabilities
0x080 Link Status Link Control
0x084 Slot Capabilities
0x088 Slot Status Slot Control
0x08C Root Capabilities Root Control
0x090 Root Status
0x094 Device Compatibilities 2
0x098 Device Status 2 Device Control 2
0x09C Link Capabilities 2
0x0A0 Link Status 2 Link Control 2
0x0A4 Slot Capabilities 2
0x0A8 Slot Status 2 Slot Control 2

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

188
A. Configuration Space Registers
UG-20225 | 2021.02.18

Figure 87. MSI-X Capability Structure

31 24 23 16 15 87 3 2 0
0x0B0 Message Control Next Cap Ptr Capability ID
MSI-X
0x0B4 MSI-X Table Offset Table BAR
Indicator
MSI-X
Pending
0x0B8 MSI-X Pending Bit Array (PBA) Offset Bit Array
- BAR
Indicator

Figure 88. PCI Express AER Extended Capability Structure

31 16 15 0
0x100 PCI Express Enhanced Capability Register
0x104 Uncorrectable Error Status Register
0x108 Uncorrectable Error Mask Register
0x10C Uncorrectable Error Severity Register
0x110 Correctable Error Status Register
0x114 Correctable Error Mask Register
0x118 Advanced Error Capabilities and Control Register
0x11C Header Log Register
0x12C Root Error Command Register
0x130 Root Error Status Register
0x134 Error Source Identification Register Correctable Error Source Identification Register

Related Information
PCI Express Base Specification 4.0

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

189
 A. Configuration Space Registers
UG-20225 | 2021.02.18

A.1.4. Physical Layer 16.0 GT/s Extended Capability Structure

Figure 89. Physical Layer 16.0 GT/s Extended Capability Structure

A.1.5. MSI-X Registers

This section describes the registers previously shown in the MSI-X capability structure.

Table 121. MSI-X Control Register

Bit Location Description Access Default Value

31 MSI-X Enable: This bit must RW 0

be set to enable the MSI-X
interrupt generation.
You need to obtain this
information from the
Configuration Intercept
Interface.

30 MSI-X Function Mask: This RW 0

bit can be set to mask all
MSI-X interrupts from this
function.
You need to obtain this
information from the
Configuration Intercept
Interface.

29:27 Reserved RO 0

26:16 Size of the MSI-X table RO Programmed via the

(number of MSI-X interrupt programming interface.
vectors). The value in this
field is one less than the size
of the table set up for this
function. Maximum value is
0x7FF (2048 interrupt
vectors).
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

190
A. Configuration Space Registers
UG-20225 | 2021.02.18

Bit Location Description Access Default Value

This field is shared among

all VFs attached to one PF.

15:8 Next Capability Pointer RO Programmed via the

Points to the PCI Express programming interface.
Capability.

7:0 Capability ID assigned by RO 0x11

PCI-SIG.

Table 122. MSI-X Table Offset Register

Bit Location Description Access Default Value

2:0 BAR Indicator Register: RO Programmed via the

Specifies the BAR programming interface.
corresponding to the
memory address range
where the MSI-X table of
this function is located (000
= VF BAR0, 001 = VF BAR1,
…, 101 = VF BAR5).
This field is shared among
all VFs attached to one PF.

31:3 Offset of the memory RO Programmed via the

address where the MSI-X programming interface.
table is located, relative to
the specified BAR. The
address is extended by
appending three zeroes to
make it Qword aligned.
This field is shared among
all VFs attached to one PF.

Table 123. MSI-X Pending Bit Array Register

Bit Location Description Access Default Value

2:0 BAR Indicator Register: RO Programmed via the

Specifies the BAR programming interface.
corresponding to the
memory address range
where the Pending Bit Array
of this function is located
(000 = VF BAR0, 001 = VF
BAR1, …, 101 = VF BAR5).
This field is shared among
all VFs attached to one PF.

31:3 Offset of the memory RO Programmed via the

address where the Pending programming interface.
Bit Array is located, relative
to the specified BAR. The
address is extended by
appending three zeroes to
make it Qword aligned.
This field is shared among
all VFs attached to one PF.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

191
 A. Configuration Space Registers
UG-20225 | 2021.02.18

A.2. Configuration Space Registers for Virtualization

A.2.1. SR-IOV Virtualization Extended Capabilities Registers Address Map

Figure 90. SR-IOV Virtualization Extended Capabilities Registers (0x230 : 0x26C)

31 24 23 20 19 16 15 0
0x230 SR-IOV Extended Capability Header Register
0x234 SR-IOV Capabilities
0x238 SR-IOV Status SR-IOV Control
0x23C TotalVFs (RO) InitialVFs (RO)
Function
0x240 RsvdP Dependency NumVFs (RW)
Link (RO)
0x244 VF Stride (RO) First VF Offset (RO)
0x248 VF Device ID (RO) RsvdP
0x24C Supported Pages Sizes (RO)
0x250 System Page Size (RW)
0x254 VF BAR0 (RW)
0x258 VF BAR1 (RW)
0x25C VF BAR2 (RW)
0x260 VF BAR3 (RW)
0x264 VF BAR4 (RW)
0x268 VF BAR5 (RW)
0x26C VF Migration State Array (RO)

A.2.2. PCIe Configuration Registers for Each Virtual Function

This section provides a description of the individual registers in the configuration space
of each virtual function (VF). To access the configuration space of VFs, refer to the
section Address Map for the User Avalon-MM Interface.

Table 124. PCIe Configuration Registers for Each Virtual Function

Address Range Register Description

0x0 : 0x3C VF PCI-Compatible Configuration Space Header Type0

0x70 : 0xA0 VF PCI Express Capability Structure

0xB0 : 0xB8 VF MSI-X Capability Structure

0x178 : 0x17C VF Alternative Routing ID (ARI) Capability Structure

0x270 : 0x27C VF TLP Processing Hints Capability Structure

0x2FC : 0x300 VF Address Translation Services Capability Structure

0x30C : 0x314 VF Access Control Services (ACS) Capability Structure

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

192
A. Configuration Space Registers
UG-20225 | 2021.02.18

A.2.2.1. Alternative Routing ID (ARI) Capability Structure

A.2.2.1.1. ARI Enhanced Capability Header Register (Offset 0x0)

Table 125. ARI Enhanced Capability Header Register

Bits Register Description Default Value Access

[15:0] PCI Express Extended 0x000E RO

Capability ID for ARI

[19:16] Capability Version 0x1 RO

[31:20] Next Capability Pointer: See description. RO

When TPH Requester Programmed via
Capability is present, points Programming Interface.
to TPH Requester Capability.
When ATS Capability is
present, but TPH Requester
Capability is not, points to
ATS Capability.
When neither TPH Requester
Capability nor ATS Capability
is present, its value is 0.

A.2.2.1.2. ARI Capability and Control Register (Offset 0x4)

The lower 16 bits of this location contain the ARI Capability Register and the upper 16
bits contain the ARI Control Register. All the fields in these registers are hardwired to
0 for all VFs.

A.2.2.2. TLP Processing Hint (TPH) Capability Structure

A.2.2.2.1. TPH Requester Enhanced Capability Header (Offset 0x0)

Table 126. TPH Requester Enhanced Capability Header

Bits Register Description Default Value Access

[15:0] PCI Express Extended 0x0017 RO

Capability ID

[19:16] Capability Version 0x1 RO

[31:20] Next Capability Pointer: Programmed via RO

Points to ATS Capability Programming Interface.
when present, NULL
otherwise.

A.2.2.2.2. TPH Requester Capability Register (Offset 0x4)

This is a read-only register that specifies the capabilities associated with the
implementation of TPH in the device.

Note: Steering Tag (ST) table must be implemented in the user logic if present. The
capability will not hold the ST table.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

193
 A. Configuration Space Registers
UG-20225 | 2021.02.18

Table 127. TPH Requester Capability Register

Bits Register Description Default Value Access

[0 ] No ST Mode Supported: 0x1 RO

When set to 1, indicates that
this Function supports the
No ST Mode for the
generation of TPH Steering
Tags. In the No ST Mode,
the device must use a
Steering Tag value of 0 for
all requests.
This bit is hardwired to 1, as
all TPH Requesters are
required to support the No
ST Mode of operation.

[1] Interrupt Vector Mode Programmed via RO

Supported: A setting of 1 Programming Interface.
indicates that the Function
supports the Interrupt
Vector Mode for TPH
Steering Tag generation. In
the Interrupt Vector
Mode, Steering Tags are
attached to MSI/MSI-X
interrupt requests. The
Steering Tag for each
interrupt request is selected
by the MSI/MSI-X interrupt
vector number.

[2] Device Specific Mode Programmed via RO

Supported: A setting of 1 Programming Interface.
indicates that the Function
supports the Device
Specific Mode for TPH
Steering Tag generation. The
client typically choses the
Steering Tag values from the
ST Table, but is not required
to do so.

[7:3] Reserved 0x0 RO

[8] Extended TPH Requester Programmed via RO

Supported: When set to 1, Programming Interface.
indicates that the Function is
capable of generating
requests with 16-bit
Steering Tags, using TLP
Prefix.

[10:9] ST Table Location: The Programmed via RO

setting of this field indicates Programming Interface.
if a Steering Tag table is
implemented for this
Function, and its location if
present.
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

194
A. Configuration Space Registers
UG-20225 | 2021.02.18

Bits Register Description Default Value Access

• 2'b00 = ST Table not

present
• 2'b01 = ST Table stored
in the TPH Requester
Capability Structure
• 2'b10 = ST values stored
in the MSI-X Table in
client RAM
• 2'b11 = reserved
Valid settings are 2'b00 or
2'b10.

[15:11] Reserved 0x0 RO

[26:16] ST Table Size: Specifies the Programmed via RO

number of entries in the Programming Interface.
Steering Tag table (0 = 1
entry, 1 = 2 entries, and so
on). The maximum table
size is 2048 entries when
located in the MSI-X table.
Each entry is 8 bits.

[31:27] Reserved 0x0 RO

A.2.2.2.3. TPH Requester Control Register (Offset 0x8)

Table 128. TPH Requester Control Register

Bits Register Description Default Value Access

[2:0 ] Steering Tag (ST) Mode: 0x0 RW

This field selects the ST
mode:
• 3'b000 = No Steering Tag
Mode
• 3'b001 = Interrupt
Vector Mode
• 3'b010 = Device-Specific
Mode
• 3'b011 - 3'b111 =
Reserved
You need to obtain this
information from the
configuration intercept
interface.

[7:3] Reserved 0x0 RO

[8] TPH Requester Enable: 0x0 RW

When set to 1, the Function
is allowed to generate
requests with TLP Processing
Hints.
You need to obtain this
information from the
configuration intercept
interface.

[31:9] Reserved 0x0 RO

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

195
 A. Configuration Space Registers
UG-20225 | 2021.02.18

A.2.2.3. Address Translation Services (ATS) Capability Structure

A.2.2.3.1. ATS Enhanced Capability Header (Offset 0x0)

Table 129. ATS Enhanced Capability Header

Bits Register Description Default Value Access

[15:0] PCI Express Extended 0x000F RO

Capability ID

[19:16] Capability Version 0x1 RO

[31:20] Next Capability Pointer: See description. RO

Points to Null Programmed via
Programming Interface.

A.2.2.3.2. ATS Capability Register and ATS Control Register (Offset 0x4)

The lower 16 bits of this location make up the ATS Capability Register, and the upper
16 bits make up the ATS Control Register.

Table 130. ATS Capability Register and ATS Control Register

Bits Register Description Default Value Access

[4:0] Invalidate Queue Depth: The 0x0 RO

number of Invalidate
Requests that the Function
can accept before throttling
the upstream connection. If
0, the Function can accept
32 Invalidate Requests.
This field is hardwired to 0
for VFs. VFs use the setting
from the parent PF’s ATS
Capability Register.

[5] Page Aligned Request: If set, 0x1 RO

indicates the untranslated
address is always aligned to
a 4096-byte boundary. This
bit is hardwired to 1.

[15:6] Reserved 0x0 RO

[20:16] Smallest Translation Unit 0x0 RO

(STU): This value indicates
to the Function the minimum
number of 4096-byte blocks
specified in a Translation
Completion or Invalidate
Request. This is a power of 2
multiplier. The number of
blocks is 2STU. A value of 0
indicates one block and a
value of 0x1F indicates 231
blocks, or 8 terabytes (TB)
total.
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

196
A. Configuration Space Registers
UG-20225 | 2021.02.18

Bits Register Description Default Value Access

This field is hardwired to 0

for VFs. VFs use the setting
from the parent PF’s ATS
Control Register.

[30:21] Reserved 0x0 RO

[31] Enable (E) bit. When Set, 0x0 RW

the Function can cache
translations.
You need to obtain this
information from
configuration intercept
interface.

A.2.2.4. Access Control Services (ACS) Capability Structure

A.2.2.4.1. ACS Extended Capability Header (Offset 0x0)

Table 131. ACS Extended Capability Header

Bits Register Description Default Value Access

[15:0] PCI Express Extended 0x000D RO

Capability ID

[19:16] Capability Version 0x1 RO

[31:20] Next Capability Pointer: See description. RO

Points to the Null Pointer Programmed via
Programming Interface.

A.2.2.4.2. ACS Capability Register (Offset 0x4)

Table 132. ACS Capability Register

Bits Register Description Default Value Access

[31:0] Capability Field 0x0 RO

A.2.2.4.3. ACS Control Register (Offset 0x6)

Table 133. ACS Control Register

Bits Register Description Default Value Access

[31:0] Control Field 0x0 RW

A.2.2.4.4. Egress Control Vector (Offset 0x8)

Table 134. Egress Control Vector

Bits Register Description Default Value Access

[31:0] Egress Control Vector 0x0 RO

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

197
 A. Configuration Space Registers
UG-20225 | 2021.02.18

A.3. Intel-Defined VSEC Capability Registers

Table 135. Intel-Defined VSEC Capability Registers (0xD00 : 0xD58)
31 : 20 19 : 16 15 : 8 7:0 PCIe Byte Offset

Next Cap Offset Version PCI Express Extended Capability ID 00h

VSEC Length VSEC Rev VSEC ID 04h

Intel Marker 08h

JTAG Silicon ID DW0 0Ch

JTAG Silicon ID DW1 10h

JTAG Silicon ID DW2 14h

JTAG Silicon ID DW3 18h

CvP Status User Configurable Device/Board ID 1Ch

CvP Mode Control 20h

CvP Data 2 24h

CvP Data 28h

CvP Programming Control 2Ch

General Purpose Control and Status 30h

Uncorrectable Internal Error Status Register 34h

Uncorrectable Internal Error Mask Register 38h

Correctable Error Status Register 3Ch

Correctable Error Mask Register 40h

SSM IRQ Request & Status 44h

SSM IRQ Result Code 1 Shadow 48h

SSM IRQ Result Code 2 Shadow 4Ch

SSM Mailbox 50h

SSM Credit 0 Shadow 54h

SSM Credit 1 Shadow 58h

A.3.1. Intel-Defined VSEC Capability Header (Offset 00h)

Table 136. Intel-Defined VSEC Capability Header
Bits Register Description Default Value Access

[31:20] Next Capability Pointer. Variable RO

Value is the starting address
of the next Capability
Structure implemented, if
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

198
A. Configuration Space Registers
UG-20225 | 2021.02.18

Bits Register Description Default Value Access

any. Otherwise, NULL. Refer

to the Configuration Address
Map.

[19:16] Capability Version. PCIe 0x1 RO

specification-defined value
for VSEC Capability Version.

[15:0] Extended Capability ID. PCIe 0x000B RO

specification-defined value
for VSEC Extended
Capability ID.

A.3.2. Intel-Defined Vendor Specific Header (Offset 04h)

Table 137. Intel-Defined Vendor Specific Header
Bits Register Description Default Value Access

[31:20] VSEC Length. Total length of 0x5C RO

this structure in bytes.

[19:16] VSEC Rev. User configurable k_vsec_rev_i RO

VSEC revision.

[15:0] VSEC ID. User configurable 0x1172 RO

VSEC ID. The default value
is 0x1172 (the Intel Vendor
ID), but you can change this
ID to your own Vendor ID.

A.3.3. Intel Marker (Offset 08h)

Table 138. Intel Marker
Bits Register Description Default Value Access

Intel Marker - An additional

marker for standard Intel
[31:0] programming software to be 0x41721172 RO
able to verify that this is the
right structure.

A.3.4. JTAG Silicon ID (Offset 0x0C - 0x18)

This read-only register returns the JTAG Silicon ID. Intel programming software uses
this JTAG ID to ensure that is is using the correct SRM Object File (*.sof).

These registers are only good for Port 0 (PCIe Gen4 x16). They are blocked for the
other Ports.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

199
 A. Configuration Space Registers
UG-20225 | 2021.02.18

Table 139. JTAG Silicon ID Registers

Bits Register Description Default Value(1) Access

[127:96] JTAG Silicon ID DW3 Unique ID RO

[95:64] JTAG Silicon ID DW2 Unique ID RO

[63:32] JTAG Silicon ID DW1 Unique ID RO

[31:0] JTAG Silicon ID DW0 Unique ID RO

A.3.5. User Configurable Device and Board ID (Offset 0x1C - 0x1D)

This register provides a user configurable device or board ID so that the user software
can determine which .sof file to load into the device.

This register is only available for Port 0 (PCIe Gen4 x16). It is blocked for the other
Ports.

Table 140. User Configurable Device and Board ID Register

Bits Register Description Default Value Access

[15:0] This register allows you to From configuration bits RO

specify the ID of the .sof
file to be loaded.

A.3.6. General Purpose Control and Status Register (Offset 0x30)

This register provides up to eight I/O pins each for Application Layer Control and
Status requirements. This feature supports Partial Reconfiguration of the FPGA fabric.
Partial Reconfiguration only requires one input pin and one output pin. The other
seven I/Os make this interface extensible.

Table 141. General Purpose Control and Status Register

Bits Register Description Default Value Access

[31:16] Reserved. N/A RO

[15:8] General Purpose Status. The 0x00 RO

Application Layer can read
these status bits. These bits
are only available for Port 0
(PCIe Gen4 x16). They are
blocked for the other Ports.

[7:0] General Purpose Control. 0x00 RW

The Application Layer can
write these control bits.
These bits are only available
for Port 0 (PCIe Gen4 x16).
They are blocked for the
other Ports.

(1) Because the Silicon ID is a unique value, it does not have a global default value.

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

200
A. Configuration Space Registers
UG-20225 | 2021.02.18

A.3.7. Uncorrectable Internal Error Status Register (Offset 0x34)

This register reports the status of the internally checked errors that are uncorrectable.
When these specific errors are enabled by the Uncorrectable Internal Error
Mask register, they are forwarded as Uncorrectable Internal Errors.

Note: This register is for debug only. Only use this register to observe behavior, not to drive
logic custom logic.

Table 142. Uncorrectable Internal Error Status Register

Bits Register Description Default Value Access

[31:13] Reserved 0x0 RO

[12] Debug Bus Interface (DBI) 0x0 RW1CS

access error status from
Config RAM block.

[11] Uncorrectable ECC error 0x0 RW1C

from Config RAM block.

[10:9] Reserved 0x0 RO

[8] RX Transaction Layer parity 0x0 RW1CS

error reported by the IP
core.

[7] TX Transaction Layer parity 0x0 RW1CS

error reported by the IP
core.

[6] Uncorrectable Internal Error 0x0 RW1CS

reported by the FPGA.

[5] cvp_config_error_latch 0x0 RW1CS

ed: Configuration error
detected in CvP mode is
reported as an uncorrectable
error. Set whenever
ssm_cvp_config_error of
the SSM Scratch CvP Status
register bit[1] rises in CvP
mode. This bit is only
available for Port 0 (PCIe
Gen4 x16), but not for the
other Ports.

[4:0] Reserved 0x0 RO

Note: The access code RW1CS represents Read Write 1 to Clear Sticky.

A.3.8. Uncorrectable Internal Error Mask Register (Offset 0x38)

This register controls which errors are forwarded as internal uncorrectable errors.

Table 143. Uncorrectable Internal Error Mask Register

Bits Register Description Default Value Access

[31:13] Reserved 0x0 RO

[12] Mask for Debug Bus 0x1 RWS

Interface (DBI) access error.
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

201
 A. Configuration Space Registers
UG-20225 | 2021.02.18

Bits Register Description Default Value Access

[11] Mask for Uncorrectable ECC 0x1 RWS

error from Config RAM block.

[10:9] Reserved 0x0 RO

[8] Mask for RX Transaction 0x1 RWS

Layer parity error reported
by the IP core.

[7] Mask for TX Transaction 0x1 RWS

Layer parity error reported
by the IP core.

[6] Mask for Uncorrectable 0x1 RWS

Internal error reported by
the FPGA.

[5] Mask for Configuration Error 0x0 RWS

detected in CvP mode. This
bit is only available for Port
0 (PCIe Gen4 x16), but not
for the other Ports.

[4:0] Reserved 0x0 RO

Note: The access code RWS stands for Read Write Sticky, meaning that the value is retained
after a soft reset of the IP core.

A.3.9. Correctable Internal Error Status Register (Offset 0x3C)

The Correctable Internal Error Status register reports the status of the
internally checked errors that are correctable. When these specific errors are enabled
by the Correctable Internal Error Mask register, they are forwarded as
correctable internal errors. This register is for debug only. Only use this register to
observe behavior, not to drive custom logic

Table 144. Correctable Internal Error Status Register

Bits Register Description Default Value Access

[31:12] Reserved 0x0 RO

[11] Correctable ECC error status 0x0 RW1CS

from Config RAM.

[10:7] Reserved 0x0 RO

[6] Correctable Internal Error 0x0 RW1CS

reported by the FPGA.

[5] cvp_config_error_latch 0x0 RW1CS

ed: Configuration error
detected in CvP mode (to be
reported as correctable) -
Set whenever
cvp_config_error rises
while in CvP mode. This bit
is only available for Port 0
(PCIe Gen4 x16), but not for
the other Ports.

[4:0] Reserved 0x0 RO

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

202
A. Configuration Space Registers
UG-20225 | 2021.02.18

A.3.10. Correctable Internal Error Mask Register (Offset 0x40)

This register controls which errors are forwarded as internal correctable errors.

Table 145. Correctable Internal Error Mask Register

Bits Register Description Default Value Access

[31:12] Reserved 0x0 RO

[11] Mask for Correctable ECC 0x1 RWS

error status for Config RAM.

[10:7] Reserved 0x0 RWS

[6] Mask for Correctable 0x1 RWS

Internal Error reported by
the FPGA.

[5] Mask for Configuration Error 0x1 RWS

detected in CvP mode. This
bit is only available for Port
0 (PCIe Gen4 x16), but not
for the other Ports.

[4] Reserved 0x1 RWS

[3:0] Reserved 0x0 RWS

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

203
 UG-20225 | 2021.02.18

Send Feedback

B. Implementation of Address Translation Services (ATS)

in Endpoint Mode
With the P-Tile Avalon streaming IP for PCIe:
• ATS messages/completions are sent and received through the Avalon streaming
interface.
• Address translation caches (ATC) need to be implemented in the user logic. There
must be a separate ATC for each VF/PF that supports ATS.

Refer to the Address Translation Services Revision 1.1 specification, section 4.1 Page
Request Message for more details.

B.1. Sending Translated/Untranslated Requests

The function with an ATC can send Memory Read requests that contain either
translated or untranslated addresses. If the Endpoint wants to receive the associated
translated address to update the ATC, it will generate a Memory Read with the
"Translation Request" field set.

The Endpoint will receive the translated address in the associated Completion with the
ATS field's S/N/G/P/E/U/W/R values.

B.2. Sending a Page Request Message from the Endpoint (EP) to the
Root Complex (RC)
The user application issues a Page Request Message while using the Avalon-ST
interface to send the contents of the ATS message.

The RC will respond with PRG Response message(s).

The EP will update its ATC accordingly.

B.3. Invalidating Requests/Completions

Invalidation is done via the Message mechanism:
• When the EP receives an Invalidate request from the RP, it needs to clear the
associated ATC.
• When the ATC is cleared, the user application generates a Completion message.

Intel Corporation. All rights reserved. Agilex, Altera, Arria, Cyclone, Enpirion, Intel, the Intel logo, MAX, Nios,
Quartus and Stratix words and logos are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or
other countries. Intel warrants performance of its FPGA and semiconductor products to current specifications in ISO
accordance with Intel's standard warranty, but reserves the right to make changes to any products and services 9001:2015
at any time without notice. Intel assumes no responsibility or liability arising out of the application or use of any Registered
information, product, or service described herein except as expressly agreed to in writing by Intel. Intel
customers are advised to obtain the latest version of device specifications before relying on any published
information and before placing orders for products or services.
*Other names and brands may be claimed as the property of others.
 UG-20225 | 2021.02.18

Send Feedback

C. Packets Forwarded to the User Application in TLP

Bypass Mode
In TLP Bypass mode, the P-Tile IP for PCIe forwards TLPs to the Avalon-ST RX
interface except for malformed TLPs. The following tables describe how the IP handles
each TLP type for upstream and downstream.

C.1. EP TLP Bypass Mode (Upstream)

Table 146. Packets Forwarded in EP TLP Bypass Mode
Forwarded to
TLP Type Routing Direction TLP Corruption
Avalon-ST Interface

None No
ASSERT/DEASSERT
Local Upstream Ecrc_err No
INTx
Malformed No

No (VENDOR0)
None
Yes (VENDOR1)

No (VENDOR0)
VENDOR_MESSAGE_0 Poisoned
Route_to_RC Upstream Yes (VENDOR1)
/1

Ecrc_err Yes

Malformed No

None Yes

ID_mismatch Yes
VENDOR_MESSAGE_0
Route_by_ID Both Poisoned Yes
/1
Ecrc_err Yes

Malformed No

None Yes

Poisoned Yes
VENDOR_MESSAGE_0
Broadcast Downstream
/1
Ecrc_err Yes

Malformed No

None Yes

Poisoned Yes
VENDOR_MESSAGE_0
Local Both
/1
Ecrc_err Yes

Malformed No

PM_ACTIVE_STATE_N
Local Downstream None Yes
AK
continued...

Intel Corporation. All rights reserved. Agilex, Altera, Arria, Cyclone, Enpirion, Intel, the Intel logo, MAX, Nios,
Quartus and Stratix words and logos are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or
other countries. Intel warrants performance of its FPGA and semiconductor products to current specifications in ISO
accordance with Intel's standard warranty, but reserves the right to make changes to any products and services 9001:2015
at any time without notice. Intel assumes no responsibility or liability arising out of the application or use of any Registered
information, product, or service described herein except as expressly agreed to in writing by Intel. Intel
customers are advised to obtain the latest version of device specifications before relying on any published
information and before placing orders for products or services.
*Other names and brands may be claimed as the property of others.
 C. Packets Forwarded to the User Application in TLP Bypass Mode
UG-20225 | 2021.02.18

Forwarded to
TLP Type Routing Direction TLP Corruption
Avalon-ST Interface

Ecrc_err Yes

Malformed No

None No

PM_PME Route_to_RC Upstream Ecrc_err Yes

Malformed No

None Yes

PME_TURN_OFF Broadcast Downstream Ecrc_err Yes

Malformed No

None No

PME_TO_ACK Gather Upstream Ecrc_err Yes

Malformed No

None No

ERR_COR Route_to_RC Upstream Ecrc_err Yes

Malformed No

None No

ERR_NONFATAL Route_to_RC Upstream Ecrc_err Yes

Malformed No

None No

ERR_FATAL Route_to_RC Upstream Ecrc_err Yes

Malformed No

None Yes

UNLOCK Broadcast Downstream Ecrc_err Yes

Malformed No

None Yes

Poisoned Yes
SET_SLOT_POWER_LI
Local Downstream
MIT
Ecrc_err Yes

Malformed No

None Yes

ID_mismatch Yes

LN_MESSAGE Route_by_ID Both Poisoned Yes

Ecrc_err Yes

Malformed No

None Yes

LN_MESSAGE Broadcast Downstream Poisoned Yes

Ecrc_err Yes
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

206
C. Packets Forwarded to the User Application in TLP Bypass Mode
UG-20225 | 2021.02.18

Forwarded to
TLP Type Routing Direction TLP Corruption
Avalon-ST Interface

Malformed No

None Yes

DRS_MESSAGE Local Upstream Ecrc_err Yes

Malformed No

None Yes

FRS_MESSAGE Route_to_RC Upstream Ecrc_err Yes

Malformed No

None Yes

Poisoned Yes
HIERARCHY_ID_MSG Broadcast Downstream
Ecrc_err Yes

Malformed No

None Yes
IGNORED_MSG_ATT_
Local Downstream Ecrc_err Yes
ON
Malformed No

None Yes
IGNORED_MSG_ATT_
Local Downstream Ecrc_err Yes
BLINK
Malformed No

None Yes
IGNORED_MSG_ATT_
Local Downstream Ecrc_err Yes
OFF
Malformed No

None Yes
IGNORED_MSG_IND_
Local Downstream Ecrc_err Yes
ON
Malformed No

None Yes
IGNORED_MSG_IND_
Local Downstream Ecrc_err Yes
BLINK
Malformed No

None Yes
IGNORED_MSG_IND_
Local Downstream Ecrc_err Yes
OFF
Malformed No

None Yes
IGNORED_MSG_ATT_
Local Upstream Ecrc_err Yes
BT_PRESS
Malformed No

None No

LTR_MESSAGE Local Upstream Poisoned No

Ecrc_err Yes
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

207
 C. Packets Forwarded to the User Application in TLP Bypass Mode
UG-20225 | 2021.02.18

Forwarded to
TLP Type Routing Direction TLP Corruption
Avalon-ST Interface

Malformed No

None No

Poisoned No
OBFF_MESSAGE Local Downstream
Ecrc_err Yes

Malformed No

None No

PTM_REQUEST Local Upstream Ecrc_err Yes

Malformed No

None No

Poisoned No
PTM_RESPONSE Local Downstream
Ecrc_err Yes

Malformed No

None No

Poisoned No
PTM_RESPONSE_D Local Downstream
Ecrc_err Yes

Malformed No

None Yes

ID_mismatch Yes
INVALIDATE_REQUES
Route_by_ID Both Poisoned Yes
T
Ecrc_err Yes

Malformed No

None Yes

ID_mismatch Yes
INVALIDATE_COMPLE
Route_by_ID Both Poisoned Yes
TION
Ecrc_err Yes

Malformed No

None Yes

ID_mismatch Yes

CFG_WR_0 Route_by_ID Downstream Poisoned Yes

Ecrc_err Yes

Malformed No

None Yes

ID_mismatch Yes
CFG_WR_1 Route_by_ID Downstream
Poisoned Yes

Ecrc_err Yes
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

208
C. Packets Forwarded to the User Application in TLP Bypass Mode
UG-20225 | 2021.02.18

Forwarded to
TLP Type Routing Direction TLP Corruption
Avalon-ST Interface

Malformed No

None Yes

ID_mismatch Yes
CFG_RD_0 Route_by_ID Downstream
Ecrc_err Yes

Malformed No

None Yes

ID_mismatch Yes
CFG_RD_1 Route_by_ID Downstream
Ecrc_err Yes

Malformed No

None Yes

Addr_mismatch Yes

IO_WR Address Downstream Poisoned Yes

Ecrc_err Yes

Malformed No

None Yes

Addr_mismatch Yes
IO_RD Address Downstream
Ecrc_err Yes

Malformed No

None Yes

Addr_mismatch Yes

MEM_WR_32/64 Address Both Poisoned Yes

Ecrc_err Yes

Malformed No

None Yes

Addr_mismatch Yes
MEM_RD_32/64 Address Both
Ecrc_err Yes

Malformed No

None Yes

Addr_mismatch Yes
MEM_RD_LK_32/64 Address Both
Ecrc_err Yes

Malformed No

None Yes

Addr_mismatch Yes
ATOMIC_FETCH_ADD_
Address Both
32/64
Poisoned Yes

Ecrc_err Yes
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

209
 C. Packets Forwarded to the User Application in TLP Bypass Mode
UG-20225 | 2021.02.18

Forwarded to
TLP Type Routing Direction TLP Corruption
Avalon-ST Interface

Malformed No

None Yes

Addr_mismatch Yes

ATOMIC_SWAP_32/64 Address Both Poisoned Yes

Ecrc_err Yes

Malformed No

None Yes

Addr_mismatch Yes

ATOMIC_CAS_32/64/1 Poisoned Yes

Address Both
28
Ecrc_err Yes

32/64: No
Malformed
128: No stimulus

None Yes

ID_mismatch Yes

LUT_mismatch Yes

Ecrc_err Yes
CPL Route_by_ID Both
Malformed No

CA_status Yes

UR_status Yes

CRS_status Yes

None Yes

ID_mismatch Yes

LUT_mismatch Yes
CPLD Route_by_ID Both
Poisoned Yes

Ecrc_err Yes

Malformed No

C.2. RC TLP Bypass Mode (Downstream)

Table 147. Packets Forwarded in RC TLP Bypass Mode
Forwarded to
TLP Type Routing Direction TLP Corruption
Avalon-ST Interface

None Yes
ASSERT/DEASSERT
Local Upstream Ecrc_err Yes
INTx
Malformed No

None Yes
VENDOR_MESSAGE_0
Route_to_RC Upstream
/1
Poisoned Yes
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

210
C. Packets Forwarded to the User Application in TLP Bypass Mode
UG-20225 | 2021.02.18

Forwarded to
TLP Type Routing Direction TLP Corruption
Avalon-ST Interface

Ecrc_err Yes

Malformed No

None Yes

ID_mismatch Yes
VENDOR_MESSAGE_0
Route_by_ID Both Poisoned Yes
/1
Ecrc_err Yes

Malformed No

None Yes

Poisoned Yes
VENDOR_MESSAGE_0
Broadcast Downstream
/1
Ecrc_err Yes

Malformed No

None Yes

Poisoned Yes
VENDOR_MESSAGE_0
Local Both
/1
Ecrc_err Yes

Malformed No

None Yes
PM_ACTIVE_STATE_N
Local Downstream Ecrc_err Yes
AK
Malformed No

None Yes

PM_PME Route_to_RC Upstream Ecrc_err Yes

Malformed No

None Yes

PME_TURN_OFF Broadcast Downstream Ecrc_err Yes

Malformed No

None Yes

PME_TO_ACK Gather Upstream Ecrc_err Yes

Malformed No

None Yes

ERR_COR Route_to_RC Upstream Ecrc_err Yes

Malformed No

None Yes

ERR_NONFATAL Route_to_RC Upstream Ecrc_err Yes

Malformed No

None Yes
ERR_FATAL Route_to_RC Upstream
Ecrc_err Yes
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

211
 C. Packets Forwarded to the User Application in TLP Bypass Mode
UG-20225 | 2021.02.18

Forwarded to
TLP Type Routing Direction TLP Corruption
Avalon-ST Interface

Malformed No

None Yes

UNLOCK Broadcast Downstream Ecrc_err Yes

Malformed No

None Yes

Poisoned Yes
SET_SLOT_POWER_LI
Local Downstream
MIT
Ecrc_err Yes

Malformed No

None Yes

ID_mismatch Yes

LN_MESSAGE Route_by_ID Both Poisoned Yes

Ecrc_err Yes

Malformed No

None Yes

Poisoned Yes
LN_MESSAGE Broadcast Downstream
Ecrc_err Yes

Malformed No

None Yes

DRS_MESSAGE Local Upstream Ecrc_err Yes

Malformed No

None Yes

FRS_MESSAGE Route_to_RC Upstream Ecrc_err Yes

Malformed No

None Yes

Poisoned Yes
HIERARCHY_ID_MSG Broadcast Downstream
Ecrc_err Yes

Malformed No

None Yes
IGNORED_MSG_ATT_
Local Downstream Ecrc_err Yes
ON
Malformed No

None Yes
IGNORED_MSG_ATT_
Local Downstream Ecrc_err Yes
BLINK
Malformed No

None Yes
IGNORED_MSG_ATT_
Local Downstream
OFF
Ecrc_err Yes
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

212
C. Packets Forwarded to the User Application in TLP Bypass Mode
UG-20225 | 2021.02.18

Forwarded to
TLP Type Routing Direction TLP Corruption
Avalon-ST Interface

Malformed No

None Yes
IGNORED_MSG_IND_
Local Downstream Ecrc_err Yes
ON
Malformed No

None Yes
IGNORED_MSG_IND_
Local Downstream Ecrc_err Yes
BLINK
Malformed No

None Yes
IGNORED_MSG_IND_
Local Downstream Ecrc_err Yes
OFF
Malformed No

None Yes
IGNORED_MSG_ATT_
Local Upstream Ecrc_err Yes
BT_PRESS
Malformed No

None Yes

LTR_MESSAGE Local Upstream Ecrc_err Yes

Malformed No

None Yes

OBFF_MESSAGE Local Downstream Ecrc_err Yes

Malformed No

None Yes

PTM_REQUEST Local Upstream Ecrc_err Yes

Malformed No

None Yes

PTM_RESPONSE Local Downstream Ecrc_err Yes

Malformed No

None Yes

Poisoned Yes
PTM_RESPONSE_D Local Downstream
Ecrc_err Yes

Malformed No

None Yes

ID_mismatch Yes
INVALIDATE_REQUES
Route_by_ID Both Poisoned Yes
T
Ecrc_err Yes

Malformed No

INVALIDATE_COMPLE
Route_by_ID Both None Yes
TION
continued...

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

213
 C. Packets Forwarded to the User Application in TLP Bypass Mode
UG-20225 | 2021.02.18

Forwarded to
TLP Type Routing Direction TLP Corruption
Avalon-ST Interface

ID_mismatch Yes

Poisoned Yes

Ecrc_err Yes

Malformed No

None Yes

ID_mismatch Yes

CFG_WR_0 Route_by_ID Downstream Poisoned Yes

Ecrc_err Yes

Malformed No

None Yes

ID_mismatch Yes

CFG_WR_1 Route_by_ID Downstream Poisoned Yes

Ecrc_err Yes

Malformed No

None Yes

ID_mismatch Yes
CFG_RD_0 Route_by_ID Downstream
Ecrc_err Yes

Malformed No

None Yes

ID_mismatch Yes
CFG_RD_1 Route_by_ID Downstream
Ecrc_err Yes

Malformed No

None Yes

Addr_mismatch Yes

IO_WR Address Downstream Poisoned Yes

Ecrc_err Yes

Malformed No

None Yes

Addr_mismatch Yes
IO_RD Address Downstream
Ecrc_err Yes

Malformed No

None Yes

Addr_mismatch Yes
MEM_WR_32/64 Address Both
Poisoned Yes

Ecrc_err Yes
continued...

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

214
C. Packets Forwarded to the User Application in TLP Bypass Mode
UG-20225 | 2021.02.18

Forwarded to
TLP Type Routing Direction TLP Corruption
Avalon-ST Interface

Malformed No

None Yes

Addr_mismatch Yes
MEM_RD_32/64 Address Both
Ecrc_err Yes

Malformed No

None Yes

Addr_mismatch Yes
ATOMIC_FETCH_ADD_
Address Both Poisoned Yes
32/64
Ecrc_err Yes

Malformed No

None Yes

Addr_mismatch Yes

ATOMIC_SWAP_32/64 Address Both Poisoned Yes

Ecrc_err Yes

Malformed No

None Yes

Addr_mismatch Yes
ATOMIC_CAS_32/64/1
Address Both Poisoned Yes
28
Ecrc_err Yes

Malformed No

None Yes

ID_mismatch Yes

LUT_mismatch Yes

Ecrc_err Yes
CPL Route_by_ID Both
Malformed No

CA_status Yes

UR_status Yes

CRS_status Yes

None Yes

ID_mismatch Yes

LUT_mismatch Yes
CPLD Route_by_ID Both
Poisoned Yes

Ecrc_err Yes

Malformed No

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

215
 UG-20225 | 2021.02.18

Send Feedback

D. Using the Avery BFM for Intel P-Tile PCI Express Gen4
x16 Simulations

D.1. Overview
This appendix describes how to set up an Intel P-Tile PCIe Gen4 x16 Endpoint
simulation using Avery BFMs for the Synopsys VCS simulator.

The Avery BFM simulation example described here is based on the PCIe Programmed
I/O (PIO) example design generated from the Intel Quartus Prime PCIe IP GUI.
Although the simulation flow and testbench setup leverage the Intel Quartus Prime
example design testbench files, a similar flow and setup can be used for other PCIe
system simulations with the P-Tile PCIe IP core.

Figure 91. Simulation Flow

Software Requirements
• Intel Quartus Prime version 20.3
• Intel P-Tile Avalon-MM/Avalon-ST PCIe IP version 3.1.0
• Avery BFM version 2.2b
• Synopsys VCS Simulator version O-2018.09-SP2-2

Simulation Script Files

The following table describes the files required for running simulations and the
locations where they need to be. Contact your local Field Applications Engineer (FAE)
to get a sample copy of these files.

Intel Corporation. All rights reserved. Agilex, Altera, Arria, Cyclone, Enpirion, Intel, the Intel logo, MAX, Nios,
Quartus and Stratix words and logos are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or
other countries. Intel warrants performance of its FPGA and semiconductor products to current specifications in ISO
accordance with Intel's standard warranty, but reserves the right to make changes to any products and services 9001:2015
at any time without notice. Intel assumes no responsibility or liability arising out of the application or use of any Registered
information, product, or service described herein except as expressly agreed to in writing by Intel. Intel
customers are advised to obtain the latest version of device specifications before relying on any published
information and before placing orders for products or services.
*Other names and brands may be claimed as the property of others.
D. Using the Avery BFM for Intel P-Tile PCI Express Gen4 x16 Simulations
UG-20225 | 2021.02.18

You can use these files as-is for Gen4 x16 PIO simulations based on the P-Tile PCIe
PIO example design.

Table 148. Simulation Script Files

File Name Description Destination Folder

apci_top_rc.sv Avery BFMs (Gen4 x16) and memory <example design folder>/pcie_ed_tb/
write & read traffic generation pcie_ed_tb/sim

pci_ed_tb.sv Top-level testbench including Avery <example design folder>/pcie_ed_tb/

BFMs and PCIe PIO example design pcie_ed_tb/sim

vcs_files.tcl List of design example simulation files <example design folder>/pcie_ed_tb/

Note: The testbench top level is a pcie_ed_tb/sim/common
System Verilog file called
pcie_ed_tb.sv.

avery_files_vcs.f Avery BFM simulation files <example design folder>/pcie_ed_tb/

pcie_ed_tb/sim/synopsys/vcs

vcs_setup.sh VCS simulation script <example design folder>/pcie_ed_tb/

pcie_ed_tb/sim/synopsys/vcs

D.2. Generate the PCIe PIO Example Design

D.2.1. Avalon-ST PCIe PIO Example Design

In the Avalon-ST PCIe IP GUI, configure the Avalon-ST PCIe IP as a Gen4 x16
Endpoint and generate the example design. This example design automatically creates
the files for simulation. Following is the top-level testbench block diagram showing the
default Root Port BFM and PIO example design.

Figure 92. PCIe PIO Example Design Simulation Testbench

For details on the example design generation, refer to the Quick Start Guide chapter of
the Intel FPGA P-Tile Avalon-ST IP for PCI Express User Guide.

D.3. Integrate Avery BFMs (Root Complex)

P-Tile supports serial mode connection only. The Avery Root Complex must instantiate
a standalone BFM PHY to interface with the P-Tile PCIe Hard IP. The Avery BFM
replaces the default Root Port BFM (dut_pcie_tb) in the P-Tile PCIe PIO example
design testbench. The following diagram is a new top-level testbench,
pcie_ed_tb.sv, that includes the Avery BFM.

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

217
 D. Using the Avery BFM for Intel P-Tile PCI Express Gen4 x16 Simulations
UG-20225 | 2021.02.18

Figure 93. Integrating the Avery BFM (Root Complex) and the PCIe Example Design

To include the Avery BFM, use the pcie_ed_tb.sv provided in the .zip file that you
download or modify the example design top-level testbench as follows:
1. Open the example design top-level testbench file <example design folder>/
pcie_ed_tb/pcie_ed_tb/sim/pcie_ed_tb.v.
2. Import Avery BFM packages by adding the following lines below the `timescale
statement:
• import avery_pkg::*;
• import apci_pkg::*;
• import apci_pkg_test::*;
• `include "apci_defines.svh"
3. Include the Avery BFM by adding the following line above the example design Root
Port BFM (dut_pcie_tb).
• `include "./../../apci_top_rc.sv"
4. Comment out the example design Root Port BFM (dut_pcie_tb).
5. Save the file and rename it to pcie_ed_tb.sv.
6. Open <example design folder>/pcie_ed_tb/pcie_ed_tb/sim/common/
vcs_files.tcl. Find and replace pcie_ed_tb.v with pcie_ed_tb.sv.

For details, refer to apci_top_rc.sv, pcie_ed_tb.sv, and vcs_files.tcl files

available in the .zip file that you download.

D.4. Configure the Avery BFM and Update the Simulation Script
Configure the Avery BFM

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

218
D. Using the Avery BFM for Intel P-Tile PCI Express Gen4 x16 Simulations
UG-20225 | 2021.02.18

In this example, the Avery BFM in the apci_top_rc.v file is configured to support
Gen4 x16 simulations as shown below:

`define APCI_NUM_LANES 16 // default: 16 lanes

rc.cfg_info.speed_sup = 4 // default: Gen4 speed

Dumping waveforms into the VPD file is also enabled in apci_top_rc.v (see
$vcdpluson() task). If you want to disable it, comment out +define+APCI_DUMP_VPD
in the avery_files_vcs.f file.

Update the Simulation Script

Before you compile/simulate the design, update the <example design folder>/
pcie_ed_tb/pcie_ed_tb/sim/synopsys/vcs/vcs_setup.sh script file to
include the Avery file and debug options (shown in bold) for the VCS command:
• vcs -lca -f avery_files_vcs.f -debug_pp -timescale=1ps/1ps …

Note: The QUARTUS_INSTALL_DIR in vcs_setup.sh points to the Quartus installation

directory. If you are using the vcs_setup.sh provided in the downloaded .zip file,
you need to update the path per your local Quartus installation.

For details, refer to apci_top_rc.v, avery_vcs_files.f, and vcs_setup.sh

provided in the downloaded .zip file.

D.5. Compile and Simulate

You can run simulations in either batch mode or interactive mode.

Note: Interactive mode is not available in the 20.3 release of Intel Quartus Prime, but may
be available in a future release.

Batch Mode

In batch mode, the VCS script compiles the design and runs the simulation until
$finish(). Run the following command under the <example design folder>/
pcie_ed_tb/pcie_ed_tb/sim/synopsys/vcs folder.

% sh ./vcs_setup.sh USER_DEFINED_SIM_OPTIONS=””

Setting USER_DEFINED_SIM_OPTIONS to an empty string overrides the default

USER_DEFINED_SIM_OPTIONS and allows the simulation to run until $finish().

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

219
 D. Using the Avery BFM for Intel P-Tile PCI Express Gen4 x16 Simulations
UG-20225 | 2021.02.18

D.6. View the Results

The Avery BFM includes a memory transaction test that runs a sequence of 10
memory write/read combinations, where the test writes to a memory location and
immediately reads back from the same location. For details on the test, refer to
mem_tr_test class in apci_top_rc.sv. When the test passes, the following is displayed:

Figure 94. Simulation Results

In the simulation results above, is_write = 1 denotes a memory write, and is_write =
0 denotes a memory read.

When you want to view the VPD waveforms, invoke the DVE GUI:

% dve

In the DVE GUI, click File -> Open Database, and select apci_top.vpd.

To add waveforms, select a component (e.g. dut) in the Hierarchy pane, select signals
in the Variable pane, and then add them to the Wave pane.

Figure 95. Sample DVE GUI Hierarchy and Variables Screen

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

220
D. Using the Avery BFM for Intel P-Tile PCI Express Gen4 x16 Simulations
UG-20225 | 2021.02.18

Figure 96. Sample Waveforms Screen

In addition, the Avery BFM enables dumping traffic into three text files to facilitate the
debugging of the transaction layer, data link layer, and physical layer functions:

In apci_top_rc.sv:

initial begin

rc = new("rc", null, APCI_DEVICE_rc, 1);

rc.port_set_tracker(-1, "tl" , 1);

rc.port_set_tracker(-1, "dll", 1);

rc.port_set_tracker(-1, "phy", 1);

end

The three tracker files that the Avery BFM generates are:
• tracker_tl_rc.txt (transaction layer TLP dump)
• tracker_dll_rc.txt (data link layer DLLP dump)
• tracker_phy_rc.txt (physical layer Ordered Set dump)

Send Feedback Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide

221
 D. Using the Avery BFM for Intel P-Tile PCI Express Gen4 x16 Simulations
UG-20225 | 2021.02.18

Here is an example of TLP transactions captured in tracker_tl_rc.txt:

Figure 97. Excerpt from tracker_tl_rc.txt

Intel® FPGA P-Tile Avalon® Streaming IP for PCI Express* User Guide Send Feedback

222