IOP : Hybrid actor-critic algorithm for quantum reinforcement learning at CERN beam lines

Schenk, Michael; Combarro, Elías F.; Li, Kevin Shing Bruce; Grossi, Michele; Vallecorsa, Sofia; Kain, Verena; Popa, Mircea-Marian

doi:10.1088/2058-9565/ad261b

Article
Report number	arXiv:2209.11044
Title	Hybrid actor-critic algorithm for quantum reinforcement learning at CERN beam lines
Author(s)	Schenk, Michael (CERN) ; Combarro, Elías F. (U. Oviedo (main)) ; Grossi, Michele (CERN) ; Kain, Verena (CERN) ; Li, Kevin Shing Bruce (CERN) ; Popa, Mircea-Marian (Bucharest, Polytechnic Inst.) ; Vallecorsa, Sofia (CERN)
Publication	2024-02-21
Imprint	2022-09-22
Number of pages	17
In:	Quantum Sci. Technol. 9 (2024) 025012
DOI	10.1088/2058-9565/ad261b (publication)
Subject category	quant-ph ; General Theoretical Physics
Accelerator/Facility, Experiment	CERN AWAKE
Abstract	Free energy-based reinforcement learning (FERL) with clamped quantum Boltzmann machines (QBM) was shown to significantly improve the learning efficiency compared to classical Q-learning with the restriction, however, to discrete state-action space environments. In this paper, the FERL approach is extended to multi-dimensional continuous state-action space environments to open the doors for a broader range of real-world applications. First, free energy-based Q-learning is studied for discrete action spaces, but continuous state spaces and the impact of experience replay on sample efficiency is assessed. In a second step, a hybrid actor-critic scheme for continuous state-action spaces is developed based on the Deep Deterministic Policy Gradient algorithm combining a classical actor network with a QBM-based critic. The results obtained with quantum annealing, both simulated and with D-Wave quantum annealing hardware, are discussed, and the performance is compared to classical reinforcement learning methods. The environments used throughout represent existing particle accelerator beam lines at the European Organisation for Nuclear Research (CERN). Among others, the hybrid actor-critic agent is evaluated on the actual electron beam line of the Advanced Plasma Wakefield Experiment (AWAKE).
Copyright/License	publication: © 2024-2025 The Author(s) (License: CC-BY-4.0) preprint: (License: CC BY 4.0)

$Reinforcement learning paradigm: the interaction between agent and environment occurs in discrete time steps. At every time step $t$, the agent observes the environment in its current state $s_t \in S$ and takes an action $a_t \in A$ following policy $\pi$. The environment transitions from state $s_t$ to $s_{t+1}$, according to the state transition probability and emits a reward $r_{t+1}$~\cite{suttBart}.$