Major-Minor Mean Field Multi-Agent Reinforcement Learning

Consider $N$ (minor) agents $i\in[N]\coloneqq\{1,\ldots,N\}$ with compact metric state and action spaces $\mathcal{X}$, $\mathcal{U}$, equipped with random states and actions $x^{i,N}_{t}$ and $u^{i,N}_{t}$ at times $t\in\mathbb{N}$, where initial states $x^{i,N}_{0}\sim\mu_{0}$ are independently sampled from some initial distribution $\mu_{0}\in\mathcal{P}(\mathcal{X})$. In addition to standard MFC, we also consider a single major agent, though the framework can be extended to multiple. Consider major agent state and action spaces, $\mathcal{X}^{0}$, $\mathcal{U}^{0}$ and state-actions $x^{0,N}_{t}$, $u^{0,N}_{t}$, with the major agent formally indexed by $i=0$. Given all actions, the agent states evolve according to kernels $p$, $p^{0}$ depending on (i) the agent’s own state-actions, (ii) the major state-actions, and (iii) the empirical MF, i.e. the $\mathcal{P}(\mathcal{X})$-valued empirical state distribution $\mu^{N}_{t}\coloneqq\frac{1}{N}\sum_{i=1}^{N}\delta_{x^{i,N}_{t}}$. This means that minor agents affect other agents only at rate $\frac{1}{N}$. In practice, we identify minor agents as all agents that matter through their MF $\mu^{N}_{t}$. Any remaining agents are major, such that the problem-specific stratification into major and minor agents is always possible.

By symmetry, the system state at any time $t$ is therefore entirely given by $(x^{0,N}_{t},\mu^{N}_{t})$. Accordingly, in MFC we share policies between all minor agents. We consider time-variant policies $\pi\in\Pi$, $\pi^{0}\in\Pi^{0}$ from some classes of major and minor policies $\Pi$, $\Pi^{0}$ that depend on an agent’s own state and $(x^{0,N}_{t},\mu^{N}_{t})$ at all times $t$. Overall, for all $i\in[N]$ and $t\in\mathbb{N}$, the finite MFC system follows

The goal is then to maximize the infinite-horizon discounted objective $J^{N}(\pi,\pi^{0})\coloneqq\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}r(x^{0,N}_{t},u^{0,N}_{t},\mu^{N}_{t})\right]$ over minor and major policies $(\pi,\pi^{0})$, with discount $\gamma\in(0,1)$ and reward function $r\colon\mathcal{P}(\mathcal{X})\to\mathbb{R}$. While an optimal behavior could be learned using standard MARL policy gradient methods, for improved tractability we introduce the following M3FC model in the case of many minor agents.

By the introduction of the MF limit, we obtain a large, more tractable subclass of cooperative multi-agent control problems, which may otherwise suffer from the curse of many agents (combinatorial joint state-action space, (Zhang et al., 2021)). We introduce the MF limit by formally taking $N\to\infty$: The finite-agent control problem is replaced by a higher-dimensional single-agent MDP – the M3FC MDP. By symmetry, we summarize minor agents into their probability law, the MF $\mu_{t}\equiv\mathcal{L}(x^{i,N}_{t})\in\mathcal{P}(\mathcal{X})$. It replaces its empirical analogue $\mu^{N}_{t}$ by a law of large numbers (LLN). Thus, by definition, the MF $\mu_{t}$ evolves forward as

with $\pi_{t}(\mu_{t})\coloneqq\pi_{t}(\cdot\mid\cdot,\mu_{t})$, product measures $\mu_{t}\otimes\pi_{t}(\mu_{t})$ of measure $\mu_{t}$ and kernel $\pi_{t}(\mu_{t})$ on $\mathcal{X}\times\mathcal{U}$, and deterministic dynamics for the MF, $T(x^{0},u^{0},\mu,h)\coloneqq\iint p(\cdot\mid x,u,x^{0},u^{0},\mu)h(\mathrm{d}x,\mathrm{d}u)$.

Therefore, the state of the limiting system consists only of the MF $\mu_{t}$ and major state $x^{0}_{t}$. As a result, we obtain the limiting M3FC MDP

with objective $J(\hat{\pi},\pi^{0})=\operatorname{\mathbb{E}}\left[\sum_{t=0}^{\infty}\gamma^{t}r(x^{0}_{t},u^{0}_{t},\mu_{t})\right]$ and transition dynamics for the MF $T(x^{0},u^{0},\mu,h)\coloneqq\iint p(\cdot\mid x,u,x^{0},u^{0},\mu)h(\mathrm{d}x,\mathrm{d}u)$. Here, we identify $\mu_{t}\otimes\pi_{t}(\mu_{t})\equiv h_{t}\in\mathcal{H}(\mu_{t})$ in the compact set $\mathcal{H}(\mu)\subseteq\mathcal{P}(\mathcal{X}\times\mathcal{U})$ of desired joint state-action distributions with first marginal $\mu$ as part of the action of the M3FC MDP.

In other words, the action of the M3FC MDP is $(h_{t},u^{0}_{t})$ where $h_{t}$ replaces all the minor agent actions by a LLN. Accordingly, minor agent policies are replaced by MFC policies $\hat{\pi}$ mapping from current $\mu_{t}$ to desired state-action distribution $h_{t}$. The limiting M3FC model abstracts away all the minor agents in the finite system, and considers only the MF and the major agents, as visualized in Figure 3. The reason for writing joint $h_{t}$ is mostly technical, as for deterministic $\hat{\pi}$, we write $\pi_{t}=\Phi(\hat{\pi}_{t})$ to reobtain agent policies $\mu_{t}$-a.e. uniquely by disintegration (Kallenberg, 2017) of $h_{t}=\hat{\pi}_{t}(\mu_{t})$ into $\mu_{t}\otimes\pi^{\prime}_{t}$ with decision rule $\pi^{\prime}_{t}\in\mathcal{P}(\mathcal{U})^{\mathcal{X}}$ and using $\pi_{t}(\mu_{t})\equiv\pi^{\prime}_{t}$. Inversely, any $\pi\in\Pi$ is represented in the MFC MDP by deterministic $\hat{\pi}_{t}=\Phi^{-1}(\pi)_{t}=\mu_{t}\otimes\pi_{t}$.

For the unfamiliar reader, in Appendix B we recap basic deterministic MFC without major agents or common noise. There, we recap Lipschitz approximation theorems and dynamic programming principles in compact spaces.

As a first step, it is well known that stationary (time-independent) policies suffice for optimality in infinite-horizon discounted MDPs. In the following, this property is also verified for the M3FC MDP. For the following technical results, we assume standard Lipschitz conditions (Gu et al., 2021; Mondal et al., 2022; Pásztor et al., 2023).

Assumption 1 is true, e.g., in finite spaces if transition matrix entries of $P$ are Lipschitz in the $|\mathcal{X}|$-dimensional MF vector. The sufficiency of stationary policies is obtained by the dynamic programming principle, which can also be used to compute exact optimal policies in the M3FC MDP. We use the value function $V^{*}$ as the fixed point of the Bellman equation, $V^{*}(x^{0},\mu)=\max_{(h,u^{0})\in\mathcal{H}(\mu)\times\mathcal{U}^{0}}r(x^{0},u^{0},\mu)+\gamma\mathbb{E}_{y^{0}\sim p^{0}(y^{0}\mid x^{0},u^{0},\mu)}V^{*}(y^{0},T(x^{0},u^{0},\mu,h))$.

The results follow from classical MDP theory (Hernández-Lerma and Lasserre, 2012). Thus, we may solve M3FC problems through the DPP, or approximately by using policy gradients with stationary policies for the M3FC MDP, which has naturally continuous actions.

Next, in order to show the approximate optimality of M3FC solutions, we first obtain propagation of chaos (Sznitman, 1991) – convergence of empirical MFs to the limiting MF. The result theoretically backs the reduction of multi-agent control to single-agent MDPs, as there is no loss of optimality in the finite problem by considering the M3FC problem. We assume standard Lipschitz conditions on policies (Gu et al., 2021; Mondal et al., 2022; Pásztor et al., 2023).

We note that Lipschitz policies are natural, as we usually parametrize policies in a Lipschitz manner; in particular, neural networks allow Lipschitz analysis (Pásztor et al., 2023; Herrera et al., 2023; Araujo et al., 2023). The result is that the limiting system approximates large finite systems.

The above motivates M3FC by the following near optimality result of M3FC MDP solutions in the finite system, as it suffices to optimize over stationary M3FC policies.

Therefore, one may solve difficult finite-agent MARL by detouring over the corresponding M3FC MDP as depicted in Figure 4, reducing to an MDP of a complexity independent of the number of agents $N$, which we solve in Section 3.

The proposed algorithm can be theoretically motivated. As shown in the following, finite-agent policy gradients (PG) estimate the true limiting M3FC MDP PG. First, note that finite state-actions $\mathcal{X},\mathcal{U}$ lead to continuous M3FC MDP actions $\mathcal{H}(\mu)$, while continuous $\mathcal{X},\mathcal{U}$ even yield infinite-dimensional $\mathcal{H}(\mu)$. Therefore, we have at least continuous MDPs, complicating value-based learning.

For this reason, we mainly consider PG methods to solve M3FC-type MARL problems. We parametrize M3FC MDP solutions via RL policies $\tilde{\pi}^{\theta}$ with parameters $\theta$, outputting $\xi\in\Xi$ from some compact parameter space $\Xi$ with a Lipschitz map $\Gamma(\xi)=\pi^{\prime}_{t}$ to $L_{\Pi}$-Lipschitz minor agent decision rules $\pi^{\prime}_{t}$ (formally, $h_{t}=\mu_{t}\otimes\pi^{\prime}_{t}$). Assuming the Lipschitzness of the policy network and its gradient in all arguments, on which there has been a great number of recent literature (see e.g. Herrera et al. (2023); Araujo et al. (2023) and references therein), we formulate Assumption 1.

Then, we can apply the PG theorem (Sutton et al., 1999) for the M3FC MDP. The M3FC MDP (3) essentially substitutes many-agent systems (1), which are natural approximations of the M3FC MDP by Theorem 2. Therefore, we show that M3FMARL (Algorithm 1) – single-agent PG on the multi-agent M3FC system – approximates the true PG of the limiting M3FC MDP, in the case of many minor agents. In other words, M3FMARL solves MARL by approximately solving the single-agent M3FC MDP using policy gradients.

Importantly, the underlying MDP complexity is independent of the number of minor agents. Therefore, we would expect Algorithm 1 to be able to perform well in M3FC-type problems, possibly compared to straightforward MARL where each agent is handled separately. Intuitively, for many agents, the reward signal for any single agent can become uninformative: A cooperative, “averaged” reward remains almost unaffected by a single agent’s actions. This well-known credit assignment issue is therefore solved by the hierarchical structure of M3FC, as credit is assigned to M3FC actions, which affect all minor agents at once and hence receive aggregated credit. Another advantage is that MFC profits from any advances in single-agent RL.

We use the proximal policy optimization (PPO) algorithm (Schulman et al., 2017) to obtain a M3FC policy $\pi_{\mathrm{RL}}$, instantiating the major minor mean field PPO (M3FPPO) algorithm as an instance of M3FMARL, Algorithm 1. Other PG algorithms (A2C, leading to M3FA2C) are also compared in our experiments. We parametrize MFs in $\mathcal{P}(\mathcal{X})$ and joint distributions in $\mathcal{H}(\mu^{N}_{t})$. In practice, for finite $\mathcal{X}$, $\mathcal{U}$, the parametrization of $\mathcal{P}(\mathcal{X})$ is immediate by finite-dimensional vectors $\mu^{N}_{t}\in\mathcal{P}(\mathcal{X})$. For M3FC actions, consider – in addition to the major agent action – the matrix $\xi\in[-1,1]^{\mathcal{X}\times\mathcal{U}}$, which is mapped to probabilities of minor actions in any minor state $\pi^{\prime}_{t}(u\mid x)\coloneqq Z^{-1}(\xi_{xu}+1+\epsilon)$, for small $\epsilon=10^{-10}$ and normalizer $Z$. For continuous $\mathcal{X}$, $\mathcal{U}$, we instead partition $\mathcal{X}$ into $M$ bins and represent $\mu^{N}_{t}$ as a histogram, mapping $\xi\in[-1,1]^{M\times 2}$ to diagonal Gaussian means and standard deviations, $\mu_{\mathcal{X}_{i}}\in\mathcal{U}$, $\sigma_{\mathcal{X}_{i}}\in[\epsilon,0.25+\epsilon]$, for each of $M$ bins $\mathcal{X}_{i}\subseteq\mathcal{X}$. Major actions $u^{0,N}_{t}$ are categorical or diagonal Gaussian as usual. For large $\mathcal{X},\mathcal{U}$, one could also consider kernel-based parametrizations (Cui et al., 2024).

We use two hidden layers of $256$ nodes and $\tanh$ activations for the neural networks of the policies. The neural network policy outputs parameters of a diagonal Gaussian over the major action $u^{0}$ and matrices $U$ as discussed above. In the discrete Beach scenario below, the neural network instead outputs a categorical distribution using a final softmax layer. We used no GPUs and around 300,000 CPU core hours on Intel Xeon Platinum 9242 CPUs. Optimal transport costs are computed using POT (Flamary et al., 2021). Our M3FC MDP implementation follows the gym interface (Brockman et al., 2016), while the implementation of multi-agent RL as in the following fulfills RLlib interfaces (Liang et al., 2018). The RL implementations in our work are based on MARLlib 1.0 (Hu et al., 2023a) (MIT license), which uses RLlib 1.8 (Liang et al., 2018) (Apache-2.0 license) with hyperparameters in Table 3, and otherwise default settings.

The M3FMARL algorithm falls into the paradigm of centralized training with decentralized execution (CTDE) (Zhang et al., 2021), as we sample a single central M3FC MDP action during training, but enable decentralized execution by sampling $\pi^{\prime}_{t}$ separately on each agent instead. For instance, when converged to a deterministic M3FC policy (of which an optimal one is guaranteed to exist by Theorem 1), the M3FC action is always trivially equal for all agents.

Since we also consider continuous minor agent action spaces in our experiments, we compare against PG methods for MARL. In particular, we firstly consider Independent PPO (IPPO), as PPO with independent learning (Tan, 1993) and parameter sharing (Gupta et al., 2017), and secondly also Multi-Agent PPO (MAPPO) with centralized critics. The latter has repeatedly shown strong state-of-the-art performance in cooperative MARL (de Witt et al., 2020; Papoudakis et al., 2021; Yu et al., 2022). We also separate major and minor agent policies for improved performance of IPPO / MAPPO. For comparison, we use the same observations for the policy input as in M3FMARL. The policy network architectures match, and the same PPO implementation and hyperparameters are shared with M3FPPO in Table 3. Minor agents are additionally allowed to observe their own states. More details can be found in Appendix R.

To verify the usefulness of M3FMARL whenever the M3FC model (1) is accurate, we consider $5$ benchmark tasks that fulfill the M3FC modelling assumptions. To begin, the simple two Gaussian (2G) problem has no major agent and is equipped with a time-dependent major state: A periodic, time-variant mixture of two Gaussians $\mu^{*}_{t}$ – the major state – is noisily observed analogously to $\mu^{N}_{t}$ via $M=49$ bins. Minor agents should then track the mixture distribution over time, which can find application for example in UAV-based cellular coverage of dynamic users (Mozaffari et al., 2016). In the Formation problem, we extend such formation control with major agents. In addition to 2G, one added major agent tracks a moving target. Meanwhile, minor agents instead track a formation around the dynamic major agent, see e.g. Yang et al. (2021) for applications. The Beach bar process is a studied classic (Arthur, 1994; Perrin et al., 2020), where minor agents minimize their distances to a bar and additionally avoid crowded areas. Here, the bar moves on a discrete torus. The Foraging problem is archetypal of swarm intelligence (Brambilla et al., 2013), and has agents forage randomly generated foraging areas. In particular, we can consider the logistics scenario depicted in Figure 1, where a major package truck moves in a restricted space (roads) while minor drones collect packages for urban parcel delivery (Marinelli et al., 2018). Drones fill up at package “foraging” areas, and unload near the major agent. Lastly, in the Potential problem, minor agents can generate a potential landscape, the gradient of which pushes the major agent – e.g., a large object affected by magnetic active matter (Jin and Zhang, 2021) – to be delivered to a variable target.

In Figure 5, we see that M3FPPO learning is stable, as M3FPPO reduces hard-to-analyze MARL to single-agent RL, avoiding pathologies of MARL such as non-stationarity of multi-agent learning, or the combinatorial complexity over numbers of agents. In Figure 6, we find similar success in directly training M3FPPO for small $N$ instead of transferring from high $N$. We conclude that M3FPPO remains applicable even with as few as $5$ agents. M3FPPO usually compares well against its A2C variant (M3FA2C) and IPPO / MAPPO, see Table 2 and Appendix R.2. Meanwhile, IPPO / MAPPO under the same hyperparameters as M3FPPO (large batch sizes, see Table 3) can be more unstable and lead to worse results, see Figure 7.