1 Introduction

The processes governing the production of heavy-flavour hadrons (those containing at least one charm or beauty quark) in hadronic collisions have recently become a focal point of study at the CERN LHC. Due to their large masses (\(m_\textrm{c,b} \gg \Lambda _\textrm{QCD}\)), heavy-flavour hadron production is calculable using perturbative quantum chromodynamics (pQCD) under the assumption of factorisation [1, 2]. In this approach, the production cross section of a given hadron species is considered as a convolution of: (i) the parton distribution functions (PDFs) of the colliding hadrons; (ii) the partonic hard-scattering cross sections for a heavy quark to be produced; and (iii) the fragmentation functions, which describe the fraction of the heavy-quark momentum carried by the heavy-flavour hadron. The fragmentation term in the calculation also governs the relative abundances of each hadron species, which are often referred to as ‘fragmentation fractions’. Typically, in the factorisation approach, the hadronisation process is taken to be independent of the collision system. Therefore, the fragmentation functions measured in \(\textrm{e}^{+}\textrm{e}^{-}\) collisions are applied equally to calculations in proton–proton (pp) collisions. The assumption of fragmentation as a universal hadronisation mechanism would also result in identical fragmentation fractions between \(\textrm{e}^{+}\textrm{e}^{-}\) and pp collisions. This has been tested extensively by measuring the production cross section ratios of different charm-hadron species.

Measurements of the \(\textrm{D}^+/\textrm{D}^\textrm{0} \) and \(\textrm{D}_{\textrm{s}}^+/\textrm{D}^\textrm{0} \) meson-to-meson production ratios in pp collisions at different energies by the ALICE Collaboration [3, 4] have shown consistent production rates of charm mesons between hadronic collisions and earlier measurements from \(\textrm{e}^{+}\textrm{e}^{-}\) and \(\textrm{e}^-\textrm{p}\) collisions [5]. However, more recent measurements of the \(\Lambda _{\textrm{c}}^{+}/\textrm{D}^\textrm{0} \) and \({\Xi _{\textrm{c}}^{0,+}}/\textrm{D}^\textrm{0} \) production ratios in pp and heavy-ion collisions [4, 6,7,8,9,10,11] have shown a marked increase in charm-baryon production with respect to charm-meson production over the \(\textrm{e}^{+}\textrm{e}^{-}\) baseline in hadronic collisions at LHC energies. This was further quantified by determining the charm-hadron fragmentation fractions in pp collisions at \(\sqrt{s} = 5.02~\text {Te}\hspace{-1.00006pt}\textrm{V} \)  [3, 4] and \(\sqrt{s} = 13~\text {Te}\hspace{-1.00006pt}\textrm{V} \)  [4]. In both cases, a significant relative enhancement of charm-baryon production was observed, along with a corresponding depletion in the charm-meson fragmentation fractions. The results are also consistent within uncertainties between \(\sqrt{s} = 5.02~\text {Te}\hspace{-1.00006pt}\textrm{V} \) and \(\sqrt{s} = 13~\text {Te}\hspace{-1.00006pt}\textrm{V} \), implying that the collision energy does not play a significant role in the relative abundances of charm-hadron species in hadronic collisions at the LHC. The increase in charm-baryon production is also reflected in an increase of the total charm production cross section in pp collisions over earlier determinations using only the measured D-meson cross sections with the fragmentation fractions from leptonic colliders [12,13,14]. More recently, efforts have also been made to extrapolate and combine the available measurements of charm hadrons in pp collisions from different LHC experiments in complementary regions of phase space in order to determine the total charm production cross section [15].

Measurements of the charm fragmentation fractions in p–Pb collisions are also of particular interest, as they reveal whether the observed changes in relative hadronisation rates for \(\textrm{e}^{+}\textrm{e}^{-}\) and pp collisions are mainly due to the increase of the particle multiplicity between these collision systems. The mean charged-particle multiplicity per unit of pseudorapidity \(\left. \left\langle \textrm{d}N_\textrm{ch}/\textrm{d}\eta \right\rangle \right| _{|\eta | < 1.0}\) in collisions at the LHC increases both as a function of energy and system size – for minimum-bias pp collisions at \(\sqrt{s} = 5.02~\text {Te}\hspace{-1.00006pt}\textrm{V} \), \(\left. \left\langle \textrm{d}N_\textrm{ch}/\textrm{d}\eta \right\rangle \right| _{|\eta | < 1.0} = 5.48\pm 0.05\;(\text {uncorr.\,syst.})\pm 0.05\;(\text {corr.\,syst.})\) [16], and for minimum-bias p–Pb collisions at \(\sqrt{s_\textrm{NN}} = 5.02~\text {Te}\hspace{-1.00006pt}\textrm{V} \), \(\left. \left\langle \textrm{d}N_\textrm{ch}/\textrm{d}\eta \right\rangle \right| _{|\eta | < 1.0}=16.81\pm 0.71\;(\mathrm{syst.})\) [17]. The ALICE Collaboration measured the \(\Lambda _{\textrm{c}}^{+}/\textrm{D}^\textrm{0} \) yield ratio in intervals of charged-particle multiplicity in pp collisions at \(\sqrt{s} = 13~\text {Te}\hspace{-1.00006pt}\textrm{V} \)  [18], showing that in a hadronic collision, even at relatively small multiplicities, charm-quark hadronisation proceeds differently from \(\textrm{e}^{+}\textrm{e}^{-}\) collisions. In the beauty sector, the measurements of \({\Lambda _{\text {b}}^{0}}\)-baryon production relative to that of B mesons at forward rapidity in pp and p–Pb collisions by the LHCb Collaboration show a modification of the fragmentation fractions among collision systems similar to that observed for charm quarks at midrapidity [19,20,21,22]. The ALICE Collaboration measured production cross sections of \(\textrm{D}^\textrm{0}\) and \(\Lambda _{\textrm{c}}^{+}\) hadrons originating from beauty-hadron decays at midrapidity in proton–proton collisions at a centre-of-mass energy \(\sqrt{s}\) = 13 TeV [23], and a baryon-to-meson enhancement similar to the one measured by LHCb at forward rapidity was observed.

The study of charm production in p–Pb collisions serves as an additional baseline to examine so-called cold nuclear matter (CNM) effects at the LHC. Due to different effects related to the presence of a nucleus in the colliding system, there may also be corresponding changes to the yield and transverse momentum (\(p_\textrm{T}\)) distribution of charm hadrons. These effects include modifications of the PDFs in nuclei (nPDFs) with respect to the proton PDFs [24, 25] or \(k_\textrm{T}\)-broadening due to multiple soft collisions before the heavy-quark pair is produced [26, 27]. The resulting modifications of the yields are typically examined using the nuclear modification factor \(R_\textrm{pPb}\)  [28]. The ALICE measurements of \(\Lambda _{\textrm{c}}^{+}\) production in pp and p–Pb collisions at \(\sqrt{s_\textrm{NN}} = 5.02~\text {Te}\hspace{-1.00006pt}\textrm{V} \)  [6] revealed a significant modification of the \(p_\textrm{T}\) distribution of the charm baryon-to-meson ratio between p–Pb and pp collisions, which was not observed for the charm meson-to-meson ratios within the current measurement uncertainties. In addition, an increase of the mean transverse momentum of \(\Lambda _{\textrm{c}}^{+}\) baryons was observed in p–Pb collisions with respect to pp collisions, with no significant difference seen for \(\textrm{D}^\textrm{0}\) mesons. This hardening of the \(p_\textrm{T}\) spectrum of \(\Lambda _{\textrm{c}}^{+}\) baryons in p–Pb collisions indicates a possible influence of radial flow [29, 30] and/or hadronisation via coalescence [31,32,33]. These effects are consistent with observations from the light-flavour sector, such as measurements of the \(\Lambda /\textrm{K}^0_\textrm{S} \) baryon-to-meson ratio in p–Pb collisions [34]. However, the \(p_\textrm{T}\)-integrated \(R_\textrm{pPb}\) of \(\Lambda _{\textrm{c}}^{+}\) baryons was found to be consistent with that of \(\textrm{D}^\textrm{0}\) mesons and with unity. With the recent addition of \({\Xi _{\textrm{c}}^{0}}\) baryons to the available measurements [7], it has become possible to study potential nuclear modifications of the total \({\textrm{c}}\overline{\textrm{c}}\) production cross section to conclude whether nPDFs affect the overall creation of charm hadrons. A recent review with a more comprehensive overview of heavy-quark hadronisation can be found in Ref. [35].

In this article, the first measurements of the total charm production cross section per unit of rapidity and the charm-hadron fragmentation fractions at midrapidity in p–Pb collisions at the LHC are presented. Section 2 outlines the previous measurements by the ALICE Collaboration that were used to compute the total charm-quark production cross section and the fragmentation fractions of charm quarks to different charm-hadron species. It also details the theoretical models that were used to extrapolate the measured charm-hadron production cross sections down to \(p_\textrm{T} =0\), where necessary. Section 3 shows the results for the charm-hadron fragmentation fractions and \({\textrm{c}}\overline{\textrm{c}}\) production cross section in p–Pb collisions at \(\sqrt{s_\textrm{NN}} = 5.02~\text {Te}\hspace{-1.00006pt}\textrm{V} \), along with the \(p_\textrm{T}\)-integrated nuclear modification factor \(R_\textrm{pPb}\). A summary is given in Sect. 4.

2 Dataset and analysis

The ALICE detector system is described in detail in Ref. [36]. The measured prompt charm-hadron production cross sections used in this article were obtained from a sample of minimum-bias p–Pb collisions collected in 2016 during Run 2 of the LHC with a total integrated luminosity of \(\mathcal {L}_\textrm{int} =287\pm 11\;\mu \textrm{b} ^{-1}\) [37]. The measurements were performed at midrapidity (defined as \(-0.96<y<0.04\) in the collision centre-of-mass frame). Prompt charm hadrons are those produced directly in the hadronisation of a charm quark or via the decay of a directly produced excited charm-hadron or charmonium state, while charm hadrons produced by the decays of beauty hadrons are denoted as ‘non-prompt’. D mesons were reconstructed in the decay channels \(\textrm{D}^\textrm{0} \rightarrow \textrm{K}^-\pi ^+\), \(\textrm{D}^+ \rightarrow \textrm{K}^-\pi ^+\pi ^+\), \(\textrm{D}_{\textrm{s}}^+ \rightarrow \phi \pi ^+\), and \(\textrm{D}^{*+} \rightarrow \textrm{D}^\textrm{0} \pi ^+\) (and respective charge conjugates) [28]. Charm baryons were reconstructed in the channels \(\Lambda _{\textrm{c}}^{+}\rightarrow \textrm{p} K^{0}_{\textrm{S}}\)  [6, 38], \(\Lambda _\textrm{c}^{+}\rightarrow \textrm{p} K^-\pi ^+\)  [38], and \({\Xi _{\textrm{c}}^{0}} \rightarrow \Xi ^-\pi ^+\) [7] (and respective charge conjugates). The \({\textrm{J}}/\psi \) mesons were reconstructed in the channel \({\textrm{J}}/\psi \rightarrow \textrm{e}^{+}\textrm{e}^{-} \) [39]. The details of the hadron reconstruction methods can be found in the corresponding citations.

In order to derive the total charm production cross section and the individual charm-hadron fragmentation fractions at midrapidity, the \(p_\textrm{T}\)-integrated rapidity-differential cross sections must be considered. For \(\textrm{D}^\textrm{0}\) mesons [28] and \(\Lambda _{\textrm{c}}^{+}\) baryons [6], the \(p_\textrm{T}\)-differential yields were measured down to \(p_\textrm{T} =0\) and the calculation of the \(p_\textrm{T}\)-integrated cross section did not require further extrapolation. For the remaining hadron species, where no measurement down to \(p_\textrm{T} =0\) is available, an extrapolation was performed based on model calculations.

For \(\textrm{D}^+\), \(\textrm{D}_{\textrm{s}}^+\), and \(\textrm{D}^{*+}\) mesons, POWHEG+PYTHIA 6 calculations were used to extrapolate the \(p_\textrm{T}\) spectra, which were measured down to \(p_\textrm{T} =1\) \(\text {Ge}\hspace{-1.00006pt}\textrm{V}/c\) for \(\textrm{D}^+\) and \(\textrm{D}^{*+}\), and \(p_\textrm{T} =2\) \(\text {Ge}\hspace{-1.00006pt}\textrm{V}/c\) for \(\textrm{D}_{\textrm{s}}^+\). POWHEG is an event generator with next-to-leading order (NLO) accuracy [40]. It is matched with PYTHIA 6 [41] to generate the parton shower and fragmentation. The CT14NLO parton distribution functions [42] with the EPPS16 parameterisation of the nPDFs [43] were used in the calculations. The POWHEG+PYTHIA 6 simulations describe the charm-meson production cross sections within uncertainties. The extrapolation factors were determined from the fraction of the theoretically calculated cross section in the visible region with respect to the \(p_\textrm{T}\)-integrated one, relying on the \(p_\textrm{T}\) shape of the production spectra without directly using the magnitude from theory. This was performed by taking the integral of the calculated production cross section for \(0<p_\textrm{T} <50\;\text {Ge}\hspace{-1.00006pt}\textrm{V}/c \) and dividing it by the corresponding integral in the measured \(p_\textrm{T}\) region for that species. The contribution to the total cross section for \(p_\textrm{T} >50\;\text {Ge}\hspace{-1.00006pt}\textrm{V}/c \) is negligible with respect to that for \(p_\textrm{T} <50\;\text {Ge}\hspace{-1.00006pt}\textrm{V}/c \), so any extrapolation to higher \(p_\textrm{T}\) values than this would have no impact on the final result. An uncertainty on this extrapolation factor was then determined by separately performing the same procedure using the following standard variations on the POWHEG+PYTHIA 6 model: (i) varying the factorisation and renormalisation scales in the POWHEG simulation (\(\mu _\textrm{r,f}\)) within the range \(0.5<\mu _{\textrm{r,f}}/\mu _0<2.0\) (where \(\mu _0 = \sqrt{m^2+p_\textrm{T} ^2}\)), with the additional constraint \(0.5<\mu _\textrm{r}/\mu _\textrm{f}<2\); and (ii) considering differences due to the uncertainty sets on the EPPS16 nPDF parametrisations. The maximum relative upward and downward differences from the central extrapolation factor provided by (i) and (ii) were added in quadrature to give the overall extrapolation uncertainty on the cross sections, which amounted to \( ^{+7.7}_{-2.2}\%\) for \(\textrm{D}^+\), \( ^{+18.8}_{-\phantom {0}7.3}\%\) for \(\textrm{D}_{\textrm{s}}^+\), and \( ^{+27.0}_{-\phantom {0}2.3}\%\) for \(\textrm{D}^{*+}\) mesons. In all cases, the contribution due to the values of \(\mu _\textrm{r,f}\) was much larger than that from the nPDF uncertainty sets. The extrapolation factors were \(1.27^{+0.10}_{-0.04}\) for \(\textrm{D}^+\) mesons, \(1.25^{+0.08}_{-0.04}\) for \(\textrm{D}^{*+}\) mesons, and \(2.08^{+0.40}_{-0.20}\) for \(\textrm{D}_{\textrm{s}}^+\) mesons.

For \({\Xi _{\textrm{c}}^{0}}\) baryons, the extrapolation was performed and reported in Ref. [7]. The POWHEG+PYTHIA 6 calculations were found to poorly describe the \(p_\textrm{T}\) shape of the measured differential cross sections of charm baryons in p–Pb collisions. Instead, for the case of \({\Xi _{\textrm{c}}^{0}}\), the quark (re)combination model (QCM) [44, 45] was chosen to perform the extrapolation, as it is tuned to reproduce previous measurements of \(\Lambda _{\textrm{c}}^{+}\)-baryon production from ALICE and provides a reasonable description of the shape of the measured \(\textrm{d}\sigma /\textrm{d}p_\textrm{T} \) of \({\Xi _{\textrm{c}}^{0}}\) baryons [7]. As the QCM model does not provide any theoretical uncertainties, the extrapolation uncertainty was defined by adding in quadrature (i) the maximum difference between the central extrapolation factor determined using QCM and those calculated using PYTHIA 8 with the Angantyr generator [46] and POWHEG+PYTHIA 6 (\( ^{+20.6}_{-\phantom {0}0.0}\%\)), and (ii) the envelope of the POWHEG+PYTHIA 6 variations for \({\Xi _{\textrm{c}}^{0}}\) as described above for the D mesons (\( ^{+23.0}_{-\phantom {0}8.6}\%\)). The resulting extrapolation uncertainty is \( ^{+30.9}_{-\phantom {0}8.6}\%\) of the total \({\Xi _{\textrm{c}}^{0}}\) production cross section at midrapidity. The final extrapolation factor for \({\Xi _{\textrm{c}}^{0}}\) baryons is \(1.74^{+0.54}_{-0.15}\). As no measurement of the \({\Xi _{\textrm{c}}^{+}}\) production cross section is available in p–Pb collisions at midrapidity, the contribution of \({\Xi _{\textrm{c}}^{+}}\) to the total charm cross section was considered as equal to that of \({\Xi _{\textrm{c}}^{0}}\) baryons, as also done in Refs. [3, 4]. This assumption was made based on isospin symmetry, and is further supported by the measurements of \({\Xi _{\textrm{c}}^{0,+}}\) in pp collisions at \(\sqrt{s} = 13~\text {Te}\hspace{-1.00006pt}\textrm{V} \)  [4], which showed that the production cross sections are consistent between the two charge states.

For \({\textrm{J}}/\psi \) mesons, the extrapolation was performed and reported in Ref. [39]. While the inclusive \({\textrm{J}}/\psi \)-meson cross section (defined as the sum of prompt and non-prompt \({\textrm{J}}/\psi \) mesons) was measured for \({p_\textrm{T} >0}\), an extrapolation was required for the non-prompt \({\textrm{J}}/\psi \) cross section to account for the \(p_\textrm{T}\) region \({0<p_\textrm{T} <1\;\text {Ge}\hspace{-1.00006pt}\textrm{V}/c}\) in order to extract the prompt cross section. The extrapolation was performed using Fixed Order plus Next-to-Leading Logarithm (FONLL) pQCD calculations [47, 48], with the standard CTEQ6.6 PDFs [49] modified according to the EPPS16 nPDF parametrisation [43] to describe p–Pb collisions. The corresponding value for the prompt component is obtained as the difference between the inclusive \({\textrm{J}}/\psi \)-meson cross section and the non-prompt one, as determined with the extrapolation procedure described above. The \(\textrm{d}\sigma /\textrm{d}y\) value for prompt \({\textrm{J}}/\psi \)-meson production from Ref. [39] was used directly in this work.

In the calculation of the total charm-hadron yield, the contribution from \(\Omega _\textrm{c}^{0}\) baryons must also be considered. Currently, no measurements of the \(\Omega _\textrm{c}^{0}\) are available in p–Pb collisions. In addition, as there is no measurement of the absolute branching ratios for any of the decay modes of the \(\Omega _\textrm{c}^{0}\) baryon, it is not yet possible to directly measure its production cross section and so it is not included in the sum of the charm-hadron cross sections. The potential contribution from \(\Omega _\textrm{c}^{0}\) baryons was instead taken into account by assuming an extreme upper limit of \(\sigma (\Omega _\textrm{c}^{0}) = \sigma ({\Xi _{\textrm{c}}^{0}})\), and assigning this as a fully asymmetric uncertainty bound on the total charm-hadron yield, as was previously done in Refs. [3, 4]. The uncertainty related to the estimated \(\Omega _\textrm{c}^{0}\)-baryon production cross section is added in quadrature to the upper extrapolation uncertainty on the total cross section. There are further contributions to the total charm cross section from doubly- and triply-charmed baryons such as \(\Xi _\textrm{cc}^{+}\) and \(\Omega _\textrm{ccc}^{++}\) as well as charmonium states that do not decay to \({\textrm{J}}/\psi \) mesons. However, these are considered to be negligible compared to the precision of the current measurement of the total cross section.

When calculating the total \({\textrm{c}}\overline{\textrm{c}}\) production cross section, additional correction factors were applied to account for the differing rapidity distributions between charm hadrons, individual charm quarks, and \({\textrm{c}}\overline{\textrm{c}}\) pairs. These were computed using the same procedure as in Refs. [3, 4]. The scaling factor between charm hadrons and single charm quarks was determined from FONLL calculations of pp collisions at \(\sqrt{s} = 5.02~\text {Te}\hspace{-1.00006pt}\textrm{V} \)  [50] and found to be unity, with a 2% uncertainty assigned based on the differences between FONLL and PYTHIA 8 calculations. The scaling between single charm quarks and charm-anticharm pairs was determined using POWHEG+PYTHIA 6 calculations. In the range \(-0.96<y<0.04\), this correction factor was found to have a value of 1.03. An uncertainty was assigned by adding in quadrature the envelopes defined by (i) variations of the \(\mu _\textrm{r,f}\) scale parameters in POWHEG and (ii) the uncertainty sets of the EPPS16 nPDFs, with a final value of \(8.7\%\). Combining these two factors, the uncertainty on the total charm cross section due to the rapidity shape is 8.9%.

In calculating the fragmentation fractions and total \({\textrm{c}}\overline{\textrm{c}}\) production cross section at midrapidity, the systematic uncertainties related to tracking, feed-down from beauty-hadron decays, \(p_\textrm{T}\) extrapolation, and luminosity were propagated as fully correlated among different charm-hadron species; all other uncertainties were propagated as uncorrelated.

3 Results

The \(p_\textrm{T}\)-integrated \(\left. \textrm{d}\sigma (\textrm{h}_\textrm{c})/\textrm{d}y\right| _{-0.96<y<0.04}\) values for all of the considered charm-hadron species in p–Pb collisions at \(\sqrt{s_\textrm{NN}} = 5.02~\text {Te}\hspace{-1.00006pt}\textrm{V} \) are listed in Table 1. For all species, the systematic uncertainties related to the extrapolation (extr.), luminosity determination (lumi.), and branching ratio (BR) are reported separately from the systematic uncertainties related to the data analysis, except where these values were combined or not applicable in their original publications.

Table 1 The \(p_\textrm{T}\)-integrated rapidity-differential cross sections for all measured charm-hadron species at midrapidity in p–Pb collisions at \(\sqrt{s_\textrm{NN}} = 5.02~\text {Te}\hspace{-1.00006pt}\textrm{V} \). For \(\Lambda _{\textrm{c}}^{+}\) (\({\textrm{J}}/\psi \)) hadrons, the published systematic uncertainties include the contributions from the BR (BR and luminosity) uncertainties

The fragmentation fractions \(f(\textrm{c}\rightarrow {\textrm{h}}_{\textrm{c}})\) were computed from the integrated production cross sections of the charm hadrons according to

$$\begin{aligned} f(\textrm{c}\rightarrow {\textrm{h}}_{\textrm{c}}) =\frac{\textrm{d}\sigma (\textrm{h}_\textrm{c})/\textrm{d} { y}|_{-0.96<{ y}<0.04}}{\sum \limits _{\mathrm {h_{c}}}{\textrm{d}\sigma ({\textrm{h}_\textrm{c}})/\textrm{d}{ y}|_{-0.96<{ y}<0.04}}}. \end{aligned}$$
(1)

In the denominator, only ground-state charm hadrons were considered. The production cross section of \({\Xi _{\textrm{c}}^{0}}\) was considered twice to represent both of the \({\Xi _{\textrm{c}}} \) charge states.The fragmentation fractions of all measured charm-hadron species in p–Pb collisions are reported in Fig. 1 (left panel) and summarised in Table 2. The results are compared with those measured previously in pp collisions at \(\sqrt{s} = 5.02~\text {Te}\hspace{-1.00006pt}\textrm{V} \) by the ALICE Collaboration. The original measurements in pp collisions published in Ref. [3] were updated in Ref. [4] by taking into account more recent measurements of the \(\Lambda _{\textrm{c}}^{+}\)-baryon cross section down to \(p_\textrm{T} =0\) and of the prompt \({\textrm{J}}/\psi \) cross section. No significant modification of the hadronisation between the two colliding systems is observed despite the larger system size and higher charged-particle multiplicity density in p–Pb collisions. The values are consistent with those measured in pp collisions at \(\sqrt{s} = 13~\text {Te}\hspace{-1.00006pt}\textrm{V} \)  [4]. The measurements are also compared with those from \(\textrm{e}^{+}\textrm{e}^{-}\) and \(\textrm{e}^-\textrm{p}\) collisions [5]. As observed for pp collisions, the fragmentation fractions in p–Pb collisions also exhibit an overall enhancement in the relative production of charm baryons and a corresponding deficit in charm-meson production with respect to \(\textrm{e}^{+}\textrm{e}^{-}\) and \(\textrm{e}^-\textrm{p}\) collisions.

Fig. 1
figure 1

Left: the fragmentation fractions for charm hadrons in p–Pb collisions at \(\sqrt{s_\textrm{NN}} = 5.02~\text {Te}\hspace{-1.00006pt}\textrm{V} \), compared with results from pp collisions at \(\sqrt{s} = 5.02~\text {Te}\hspace{-1.00006pt}\textrm{V} \)  [3, 4] and from \(\textrm{e}^{+}\textrm{e}^{-}\) and \(\textrm{e}^-\textrm{p}\) collisions at lower energies [5]. The fragmentation fractions for \({\textrm{J}}/\psi \) are multiplied by 20 for visibility. Right: total \({\textrm{c}}\overline{\textrm{c}}\) production cross section per unit of rapidity and per nucleon in p–Pb collisions, compared with measurements in pp collisions from ALICE [3, 4], PHENIX [51], and STAR [52], and with FONLL [47, 48] and NNLO [53,54,55] pQCD calculations. The statistical uncertainties are shown as bars, and the systematic uncertainties as unfilled boxes

Table 2 Fragmentation fractions \(f(\textrm{c}\rightarrow {\textrm{h}}_{\textrm{c}})\) of charm hadrons in pp [4] and p–Pb collisions at \(\sqrt{s_\textrm{NN}} = 5.02~\text {Te}\hspace{-1.00006pt}\textrm{V} \)

The total \({\textrm{c}}\overline{\textrm{c}}\) production cross section at midrapidity was computed as the sum of the measured \(p_\textrm{T}\)-integrated cross sections of the ground-state charm hadrons: \(\textrm{D}^\textrm{0}\), \(\textrm{D}^+\), \(\textrm{D}_{\textrm{s}}^+\), \(\Lambda _{\textrm{c}}^{+}\), \({\Xi _{\textrm{c}}^{0}}\), and \({\textrm{J}}/\psi \). The \({\Xi _{\textrm{c}}^{0}}\) contribution was considered twice to account for the \({\Xi _{\textrm{c}}^{+}}\) cross section, as discussed above. The rapidity shift between the measured charm hadrons and the originating charm-quark pairs was accounted for by multiplying the sum of the hadron species by the aforementioned correction factor of 1.03. The correlations of the systematic uncertainties between charm-hadron species were treated in the same way as for the fragmentation fractions discussed above. The resulting charm cross section is

$$\begin{aligned}&\left. \frac{\textrm{d}\sigma ({\textrm{c}}\overline{\textrm{c}})}{\textrm{d}y}\right| ^{\text {p--Pb},\sqrt{s_\textrm{NN}} = 5.02~\text {Te}\hspace{-1.00006pt}\textrm{V} }_{-0.96<y<0.04} = 219.6 \pm 6.3\;(\mathrm {stat.}) _{-11.8}^{+10.5}\;\nonumber \\ &(\mathrm {syst.}) _{-2.9}^{+8.3}\;(\mathrm {extr.}) \pm 5.4\; (\textrm{BR}) \pm 4.6\;(\mathrm {lumi.}) \pm 19.5\;\nonumber \\ &(\text {rapidity shape})+15.0\;(\Omega _\textrm{c}^{0})~\textrm{mb}. \end{aligned}$$
(2)

The measured cross section is compared to results in pp collisions at other centre-of-mass energies from ALICE [3, 4], PHENIX [51], and STAR [52] in Fig. 1 (right). In order to be directly comparable with the pp results, the cross section measured in p–Pb collisions is scaled down by a factor 1/A with \(A=208\), under the assumption that the charm production cross section in proton–nucleus collisions scales with the mass number of the larger nucleus due to the binary scaling of hard processes with the number of nucleon–nucleon collisions. The charm production cross section per nucleon in p–Pb collisions at \(\sqrt{s_\textrm{NN}} = 5.02~\text {Te}\hspace{-1.00006pt}\textrm{V} \) is consistent with the one measured in pp collisions at \(\sqrt{s} = 5.02~\text {Te}\hspace{-1.00006pt}\textrm{V} \), and is also in agreement with the general trend of results across different collision energies. The results are compared with pQCD calculations in the FONLL [47, 48] and Next-to-Next-to-Leading Order (NNLO) [53,54,55] approaches. As observed in previous works relating to the charm production cross section in pp collisions, while the experimental results tend to lie at the upper edge of the theoretical uncertainties, the pQCD approaches describe the evolution of the \({\textrm{c}}\overline{\textrm{c}}\) production cross section as a function of \(\sqrt{s_{\textsc {NN}}}\).

The \(p_\textrm{T}\)-integrated nuclear modification factor \(R_\textrm{pPb}\) for \({\textrm{c}}\overline{\textrm{c}}\) production in p–Pb collisions at \(\sqrt{s_\textrm{NN}} = 5.02~\text {Te}\hspace{-1.00006pt}\textrm{V} \) was computed from the total charm cross section discussed above. The \(R_\textrm{pPb}\) is calculated from the measured cross sections in p–Pb and pp collisions at the same centre-of-mass energy as

$$\begin{aligned} R_\textrm{pPb} =\frac{1}{A}\frac{\left. \textrm{d}\sigma /\textrm{d}y\right| ^{\textrm{pPb}}_{-0.96<y<0.04}}{\left. \textrm{d}\sigma /\textrm{d}y\right| ^{\textrm{pp}}_{|y|<0.5} \times \alpha _y}, \end{aligned}$$
(3)

with \(A=208\) as the Pb-nucleus mass number. A value of \(R_\textrm{pPb} =1\) would imply that there is no modification of the total yield at midrapidity in p–Pb collisions with respect to pp collisions. The correction factor \(\alpha _y=1.01\) accounts for the different y coverage of the measurements in pp and p–Pb collisions due to the centre-of-mass rapidity shift between the two collision systems. It was estimated from FONLL calculations of the \(\textrm{d}\sigma /\textrm{d}y\) distribution of charm quarks. The uncertainties on the measured charm cross sections in pp and p–Pb collisions due to the branching ratios and the contribution from \(\Omega _\textrm{c}^{0}\) baryons were treated as fully correlated in the ratio, and the rapidity shape uncertainties as partially correlated. All other sources of uncertainty were propagated as fully uncorrelated.

Fig. 2
figure 2

The \(p_\textrm{T}\)-integrated nuclear modification factor \(R_\textrm{pPb}\) for \(\textrm{D}^\textrm{0}\) mesons [28], \(\Lambda _{\textrm{c}}^{+}\) baryons [6], \({\textrm{J}}/\psi \) mesons [39], and \({\textrm{c}}\overline{\textrm{c}}\) pairs in p–Pb collisions at \(\sqrt{s_\textrm{NN}} = 5.02~\text {Te}\hspace{-1.00006pt}\textrm{V} \). The results are compared with calculations using the nCTEQ15\(_{\textrm{rwHF}}\) [56, 57] (yellow shaded band) and EPPS16 [43] (blue crosshatched band) nPDF sets. The horizontal lines represent the central values of the respective predictions

The \(R_\textrm{pPb}\) for \({\textrm{c}}\overline{\textrm{c}}\) pair production at midrapidity at \(\sqrt{s_\textrm{NN}} = 5.02~\text {Te}\hspace{-1.00006pt}\textrm{V} \) is shown in Fig. 2 and is

$$\begin{aligned} & \left. R_\textrm{pPb} \right| ^{-0.96<y<0.04}_{p_\textrm{T} >0}({\textrm{c}}\overline{\textrm{c}}) = 0.91 \pm 0.04\;\mathrm{(stat.)} ^{+0.08}_{-0.09}\; \nonumber \\ & \qquad \quad \mathrm{(syst.)} ^{+0.05}_{-0.03}\;\mathrm{(extr.)} \pm 0.03\;\mathrm{(lumi.)}, \end{aligned}$$
(4)

which is consistent with unity within 1\(\sigma \). This implies that the overall charm production rate in p–Pb collisions is consistent with a binary scaling of pp collisions without any significant modification from nuclear effects.

The result is also compared with those for \(\textrm{D}^\textrm{0}\) mesons [28], \(\Lambda _{\textrm{c}}^{+}\) baryons [6], and \({\textrm{J}}/\psi \) mesons [39]. The \(p_\textrm{T}\)-integrated \(R_\textrm{pPb}\) values are consistent within uncertainties between all species, implying that the individual particle species’ production rates at midrapidity are similarly unaffected by nuclear effects. The results are compared with calculations of the \(p_\textrm{T}\)-integrated \(\textrm{D}^\textrm{0}\) \(R_\textrm{pPb}\) using both the EPPS16 [43] and nCTEQ15\(_{\textrm{rwHF}}\) [56, 57] nPDF sets. In the latter case, the nCTEQ15 predictions are modified by a Bayesian reweighting, constrained to heavy-flavour hadron measurements from the LHC [57]. The predictions of the \(p_\textrm{T}\)-integrated \(R_\textrm{pPb}\) are considered to be equivalent for all charm-hadron species, as only the parton distribution functions are varied between the pp and p–Pb calculations, and the \(p_\textrm{T}\)-integrated fragmentation functions cancel in the ratios. The calculation using EPPS16 is consistent with unity within large uncertainties, while the nCTEQ15 nPDFs predict a slight suppression of charm production. The experimental results are described by both of the calculations within uncertainties. In the case of calculations with nCTEQ15, the experimental results are on the upper edge of the model uncertainty.

4 Summary

The total charm production cross section per unit of rapidity and charm-hadron fragmentation fractions were measured for the first time at midrapidity in p–Pb collisions at \(\sqrt{s_\textrm{NN}} = 5.02~\text {Te}\hspace{-1.00006pt}\textrm{V} \). The fragmentation fractions for all charm-hadron species were found to be consistent for pp and p–Pb collisions at the same collision energy, indicating that there is no significant modification of the charm-quark hadronisation process due to the different hadronic collisions’ system sizes. The total charm production cross section per nucleon in p–Pb collisions was measured to be consistent with pp collisions at the same centre-of-mass energy within the uncertainties, meaning that the assumption of binary scaling holds for charm-quark production. This was further confirmed by examining the nuclear modification factor \(R_\textrm{pPb}\), which has a value consistent with unity at midrapidity and compatible with both the individual \(p_\textrm{T}\)-integrated charm-hadron \(R_\textrm{pPb}\) results and with model calculations including nuclear modification of the PDFs. Future measurements during Run 3 of the LHC will allow for a reduction of the uncertainties possibly reducing the reliance on model-dependent extrapolations. A particular focus in this field will be the determination of charm and beauty fragmentation fractions in Pb–Pb collisions, in order to precisely quantify modification of hadronisation driven by the formation and presence of a quark–gluon plasma, like modification of the abundance of different charm-hadron species [58,59,60,61] and quarkonium states [62, 63]. Future measurements in p–Pb collisions with higher precision will further allow significant constraints to be drawn on nuclear modifications of the PDFs.