Axion-like particles (ALPs) decaying into photons are known to affect a wide range of astrophysical and cosmological observables. In this study we focus on ALPs with masses in the keV–MeV range and lifetimes between 104 and 1013 seconds, corresponding to decays between the end of Big Bang Nucleosynthesis and the formation of the Cosmic Microwave Background (CMB). Using the CosmoBit module of the global fitting framework GAMBIT, we combine state-of-the-art calculations of the irreducible ALP freeze-in abundance, primordial element abundances (including photodisintegration through ALP decays), CMB spectral distortions and anisotropies, and constraints from supernovae and stellar cooling. This approach makes it possible for the first time to perform a global analysis of the ALP parameter space while varying the parameters of ΛCDM as well as several nuisance parameters. We find a lower bound on the ALP mass of around ma > 300 keV, which can only be evaded if ALPs are stable on cosmological timescales. Future observations of CMB spectral distortions with a PIXIE-like mission are expected to improve this bound by two orders of magnitude.