FPGA roadmap

and some BPM status

Kaj Rosengren
FPGA Designer – Beam Diagnostics

www.europeanspallationsource.se
1 August 2024
Agenda

• FPGA development roadmap

– Comparison with last year
– FPGA framework progress
– Data-on-demand status
• FPGA systems roadmap
• BPM firmware status

2
FPGA Roadmap - 2018

3
FPGA Roadmap - 2018

4
FPGA Developers forum

• Started in April this year

• https://confluence.esss.lu.se/display/HAR/FPGA+Developer+Forum
• Coding guidelines, directory structure etc is in place.
– They are pretty much agreed.
– Gitlab as official repository (BI firmware projects have yet to move)
• Continuous integration has been setup in gitlab by ICS.
– Using gitlab runners to upload results to https://artifactory.esss.lu.se/
• The architecture discussions are ongoing. Mainly how to deal with IOxOS.
– Many complaints about Tosca, but no real action taken…
– Will come back to this in the framework slides

5
FPGA Roadmap - 2018

6
AXI FPGA Framework

• V0.3 was just a stabilization release.

– Still uses the Struck DMA
– It’s the version LLRF, BCM, BPM is using at the moment.
• V1.0 is ongoing and is using the Xilinx DMA
– Gives us complete freedom of defining registers.
– It also has good performance, but that more of a positive
side-effect.

7
AXI FPGA Framework – SIS8300

8
AXI FPGA Framework – Register Map

• 0x00_0000 - Framework registers

– Identification of features and support packages
– Shared components, many can be enabled/disabled by constants.
– Generic acquisition (data channels, trigger selection)
• 0x40_0000 - Board support package registers
– Board component configuration. Microblaze can be used as “abstraction”.
• 0x80_0000 - RTM support package registers (if board has RTM)
• 0x90_0000 - FMC0 support package registers (if board has FMCs)
• 0xA0_0000 - FMC1 support package registers (if board has FMCs)
• 0xC0_0000 - Application registers

9
AXI FPGA Framework – Register map

• Developed by Christian. Available here: https://

confluence.esss.lu.se/display/RFG/Register+Bank+Generator
• Script currently generates VHDL AXI4-lite register banks.
• The definitions can be re-used in python:
import sis8300ku_bsp_regbank
board_under_test = essffw.Device(essffw.SIS8300XilDrv())
board_under_test.add_register_bank(sis8300ku_bsp_regbank.sis8300ku_bsp_regbank)

passed_tests = board_under_test.test_registers_after_reset()
board_under_test["BSP_CLK_DISTR_MUX"]["CLK_SEL"].write(0x0)
• Has worked pretty flawlessly.
• But there are a lot of features that we could use to improve

10
AXI FPGA Framework – Register map

• Missing field definitions, descriptions.

• More targets of generation
– C Header files
– Confluence or other documentation output
• Need support of hierarchical inclusion. I.e. generate one register map and re-use it
multiple times (needs workaround at the moment).
• Currently the register definition is done directly in python.
– Hinko has been exploring the usage of TOML “Tom's Obvious, Minimal Language” as an alternative
for defining registers.
– There are similar systems available. Example from CERN: https://gitlab.cern.ch/cohtdrivers/cheby

11
AXI FPGA Framework – SIS8300

Generic firmware pretty much

complete. What is left:
• Some board component
control.
• Microblaze code stability
• Driver/EPICs development
• Test multiple systems in
single crate

DMA performance (transfer

2MB or larger):
• WR: ~2.8 GB/s
• RD: ~2.2 GB/s

12
Struck SIS8160

• Low-pin count FMC carrier from

Struck
• Made for high-speed applications so
not a “generic” carrier.
• Is using a similar firmware
configuration as sis8300ku.
• Decided to use this as a proof-of-
concept of the AXI FPGA framework
portability.
• Mounted CAENels FMC-Pico-1M4
for use in APT/GRID

13
AXI FPGA framework – sis8160

14
AXI FPGA framework – sis8160

• Received the Struck example firmware of the

board in July.
– Roshan, student from LTH (not working full time)
started porting it to the framework and exploring
how pico integration would look like.
• End of august. Project selecting two pico
FMCs can be built in the framework and read
data read from registers (using Struck driver)
• Mid September started ported the project to
Xilinx DMA
– Basically whole BSP built of Xilinx components in
block diagrams.
– Microblaze + I2C hooked up to FMCs to read
calibration coefficients.
– Board flashing same as sis8300.

15
AXI FPGA framework – sis8160

• Generic acquisition is complete. Features

missing:
– Setup of board components (not needed for
PICO application)
– As 8300ku need driver/EPICS development.
But can be shared.

16
AXI framework – IFC1410 (suggestion)

17
AXI FPGA Framework – IFC1410

• Goal would be to try to get rid of as

much of Tosca as possible.
– Removal of “XUSER_support” parts
of Tosca.
– Control FMCs through framework
code. Would need porting of FMC
control into framework.
• Another question mark is SW
sharing between Xilinx DMA and
Tosca2 versions.
• Of course the preferred solution
would be if we could get rid of it all
together…

18
FPGA Roadmap - 2018

19
Verification topics from last roadmap

Simulation
• Evaluate the simulator situation.
– FPGA Forum has decided to get more Questa licenses
• Ability to run regressions.
• TB templates – How-to deal with System models and golden references
• Assertions (PSL or SystemVerilog) – could at least be used for simulation
of standard interfaces.
– Neither of these topics have gotten any attention yet
• ”top-level” test benches – would be nice to have
– Xilinx DMA solution comes with example testbench. So we now have chip-level
testbenches for sis8300/8160
– Currently its only used for sanity checks. On questasim testbench runs fine, on
Vivado simulator terribly slow.
– Could quite easily generate register test from python. Not a priority though.

20
FPGA Roadmap - 2018

21
Reminder – normal PV procesing

• The ”Normal” case is that the IOC clients are updating small
PV:s with heavily processed items or shorter waveforms
• But we do acquire a lot more data than this and it would be
very useful to have it available for analysis.
22
Reminder – Data-on-demand

• The digitizer/CPU we use have quite a bit of DDR.

• Store incoming data (from RAW input channel or processed data)
into circular buffer.
• On a “trigger” read the data from the digitizer and move it into
CPU memory. Where it could be published as a PV or stored in
local storage.
23
Reminder of use-case

24
Current status

• Systems implementing circular buffering

– nBLM, icBLM, BPM, BCM
– Will also be available for RAW data in generic firmware

• Timing system integration and EPICs layers needs work.

– Lab setup with multiple systems connected with timing system that
can generate DOD events.
– BPM/BCM circular buffering needs SW support, only tested in python
scripts.
– All systems needs EPICS integration to support read-out on events.

25
Data-on-demand – EPICS layer

26
Data-on-demand - Timestamping

• Once an event has been detected and waveforms read out

these are published as PV arrays and timestamp attached.
• Timestamps for the data in the circular buffers needs to
stored as well:
– icBLM/nBLM SW approach (SW buffering)
– BCM/BPM HW approach (timestamps pushed to firmware)

27
FPGA roadmap - 2019

28
FPGA Systems roadmap

• Started a FPGA systems summary page here: https://

confluence.esss.lu.se/display/PBITECH/FPGA

29
BPM Firmware status

• BPM MEBT Version

– 9U Crate
– 1 x 1000W Wiener power supply
– 1 x NAT MCH
– 1 x Concurrent Tech AM 900/412 CPU
– 4 x SIS8300KU + 4 DWC8300 RTMs
– 1 x EVR-300
– In total 8 BPM inputs for MEBT
• MEBT version of firmware/IOC has
been deployed.
– No Data-on-demand support in IOC
– Running on Timing system loop-back
cable.

30
BPM Firmware

• Based on BPM firmware from Fredrik Kristensen, the firmware itself its
inherited from LLRF.
– Firmware was basically “duct taped” into the demo firmware from Struck.
– The processed data channels where interleaved into the default Struck output channels.
– The main problem with the firmware version is that parsing the waveforms and
publishing on PVs consumes a lot of CPU.
• Development started last autumn, main items to fix
– Port to ESS FPGA framework (IQ sampling in framework)
– Waveform channels with data conversion, gain/offset correction, decimation.
– Averaging of region-of-interest within waveform for pulse X/Y/PHASE/MAG
• Some late changes
– Add second reference line.
– Change the back-plane trigger selection to be more flexible.

31
BPM Firmware overview

32
 BPM Firmware - components

• Goal is to have components that ”plugs” somewhat simply into the framework.
- Register bank generated from scripts, note that the current version has a “Struck” version
- Components can be added in BD, register banks connected with AXI-lite interconnect (and decoding).
More investigation needed to how we handle this in the register bank scripts.
- Components can be identified through registers. Version, which features are enabled etc.

33
BPM firmware – components

• Components should have:

– Preferably some documentation in confluence.
– Testbench that tries the main configuration. It should be modifiable for other applications.
– Be IP packaged for Vivado. (still todo)
• Currently mainly small components or larger ones are planned for the “common”
components.
• HLS components is another question mark.

34
Closing

Overall BI FPGA strategy

– We are buying into Xilinx components/tools. No change in foreseeable
future.
– We want to be open source. The plan is to eventually open the
framework and components to everyone.
– We don’t have enough manpower for larger IP developments (and we
don’t have a lot of money..). So we have to do a bit of thinking about:
• Optical link development (Robustness, more generic)
• JESDB204C/B
• GigE Vision camera.
– Collaboration with other institutes.

35
Thank you!

• Questions?

36