Radiation Measurements 156 (2022) 106822

Contents lists available at ScienceDirect

Radiation Measurements
journal homepage: www.elsevier.com/locate/radmeas

Lessons learnt from the recent EURADOS intercomparisons in

computational dosimetry
Hans Rabus a, g, *, Maria Zankl b, g, José Maria Gómez-Ros c, g, Carmen Villagrasa d, g,
Jonathan Eakins e, g, Christelle Huet d, g, Hrvoje Brkić f, g, Rick Tanner e, g
a
Physikalisch-Technische Bundesanstalt (PTB), Abbestrasse 2-12, 10587, Berlin, Germany
b
Helmholtz Zentrum München German Research Center for Environmental Health (HMGU), Neuherberg, Germany
c
Centro de Investigaciones Energéticas, Medioambientales y Tecnológicas (CIEMAT), Madrid, Spain
d
Institut de Radioprotection et de Sûreté Nucléaire (IRSN), Fontenay-aux-Roses, France
e
UK Health Security Agency (UKHSA), Didcot, United Kingdom
f
J. J. Strossmayer University of Osijek (MEFOS), Osijek, Croatia
g
European Radiation Dosimetry Group (EURADOS) e.V, Neuherberg, Germany

A B S T R A C T

Organized by Working Group 6 “Computational Dosimetry” of the European Radiation Dosimetry Group (EURADOS), a group of intercomparison exercises was
conducted in which participants were asked to solve predefined problems in computational dosimetry. The results of these comparisons were published in a series of
articles in this virtual special issue of Radiation Measurements. This paper reviews the experience gained from the various exercises and highlights the resulting
conclusions for future exercises, as well as regarding the state of the art and the need for development in terms of quality assurance for computational dosimetry
techniques.

1. Introduction of the art in applied computational dosimetry, as well as conclusions for

future exercises.
The European Radiation Dosimetry Group (EURADOS) is an associ­
ation of 80 institutions and more than 600 individual members that 2. Overview of the exercises
promotes harmonization and good practice in dosimetry (Rühm et al.,
2018, 2020; Harrison et al., 2021). EURADOS has eight working groups The exercises can be divided roughly into two classes depending on
dealing with different aspects and application areas of radiation the nature of their solutions. The first class comprised six exercises on
dosimetry. EURADOS Working Group 6 on “Computational Dosimetry” the use of ICRP computational reference phantoms (ICRP, 2009; Zankl
has a cross-cutting character. Its main activities include the organization et al., 2021b) and one on unfolding of neutron spectra from Bonner
of comparison exercises (Tanner et al., 2004; Gualdrini et al., 2005, sphere measurements (Gómez-Ros et al., 2022). The former will be
2008; Siebert et al., 2006; Price et al., 2006; Broggio et al., 2012; Vrba referred to as “voxel-phantom exercises” throughout this article, the
et al., 2014, 2015; Caccia et al., 2017) and training courses (Rabus et al., latter as the “Bonner sphere exercise”. Despite the quite different nature
2021a) as well as studies on fundamental aspects of computational of the problems to be solved, these exercises had in common that they
dosimetry. required the application of well-known methodologies and established
Recently, several computational dosimetry exercises have been computational tools. This allowed the organizers to establish prior
completed, the results of which are compiled in this virtual special issue reference solutions that could be used to validate the results subse­
of Radiation Measurements (De Saint-Hubert et al., 2021, 2022; Eakins quently submitted by the participants in the exercise.
et al., 2021; Gómez-Ros et al., 2021, 2022; Huet et al., 2022; Rabus et al., The second class are exercises where no reference solutions could be
2021b; Villagrasa et al., 2022; Zankl et al., 2021a, 2021b, 2021c). This established, since one of the objectives of the exercises was evaluating
current article is meant as a synopsis and reflection on the common is­ the possible influence of different cross-section models in the codes used
sues found in those different exercises and the lessons learnt on the state by the participants. Two of these exercises were code intercomparisons,

* Corresponding author. Physikalisch-Technische Bundesanstalt (PTB), Abbestrasse 2-12, 10587, Berlin, Germany.
E-mail address: [email protected] (H. Rabus).

https://doi.org/10.1016/j.radmeas.2022.106822
Received 15 May 2022; Accepted 11 June 2022
Available online 22 June 2022
1350-4487/© 2022 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
H. Rabus et al. Radiation Measurements 156 (2022) 106822

one dealing with the calculation of microdosimetric and nanodosimetric were to be determined as well as S-values for the resulting source and
quantities (Villagrasa et al., 2019, 2022), and the other with the effects target organ combinations for two specific radionuclides.
of gold nanoparticles on dose deposition at the microscopic scale (Li
et al., 2020a, 2020b; Rabus et al., 2021b, 2021c); the former is called the 2.2. Bonner sphere spectra unfolding exercise
“uncertainty exercise” in this article and the latter the “nanoparticle
exercise”. The other two exercises in this class dealt with out-of-field The tasks were defined by the counts measured by a set of twelve
dose calculations. One was about calculating the dose to the foetus Bonner spheres of different diameters and known sensitivities (as
during maternal proton therapy treatment and the other was about determined by the organizers with radiation transport simulations),
calculating the secondary neutron fluence. (De Saint-Hubert et al., 2021, located at a measurement point in one of four known environments:
2022). These two exercises are referred to as “foetus dose exercises”. inside the bunker of a medical linac; near a radioactive source; in a
It is important to note that none of the exercises was intended to be a simulated workplace field within a neutron calibration facility; or
code competition. Rather, the aim was to investigate the dispersion of outside a nuclear power plant. The count rates measured by the Bonner
results when the same problem was solved by different people using spheres were determined by the organizers through Monte Carlo radi­
different approaches and different codes or the same code with different ation transport simulations of the respective complete measurement
options. The first class of exercises focused on identifying the state of the setup for each Bonner sphere within its environment. In addition, and to
art in the application of common methods in computational dosimetry. recreate a realistic situation, the count rate from one of the Bonner
The second class was more exploratory in nature and aimed to assess the spheres in one of the scenarios was intentionally given an incorrect value
state of the art in terms of the capabilities of codes and approaches. All in order to test the participants’ ability to detect an erroneous mea­
classes contained tasks of different complexity, and thus different de­ surement and exclude it when applying the deconvolution procedure
mands on the participants’ skills. (Gómez-Ros et al., 2018, 2022).
The exercises were organized and run by ad-hoc teams composed of
EURADOS WG 6 members. In general, the preparation of the exercises 2.3. Micro- and nanodosimetric uncertainty exercise
involved independent simulations by several team members, with their
respective results then cross-referenced to identify potential pitfalls in In its first phase, this exercise included a microdosimetric and a
the proposed exercise definitions and to check whether the tasks were nanodosimetric intercomparison (Villagrasa et al., 2019). In the frame of
solvable based on the information to be provided. For the exercises with the former, frequency distributions of specific energy were to be deter­
reference solutions, the results of these test simulations were also used to mined within a microscopic water sphere for different distributions of a
establish those values, (e.g. by taking the mean), as well as gain a handle low-energy electron emitter with an energy spectrum derived from the
on the typical levels of uncertainty that may be considered acceptable internal-conversion Auger emitter 125I. In the nanodosimetry part,
for them. ionization cluster size distributions were to be determined in target
In some exercises, templates for reporting results were also provided spheres of different sizes located at different distances from a point
to the participants. source with the same energy spectrum as in the microdosimetry part. A
sensitivity analysis was also performed on the variation of inelastic
2.1. ICRP reference voxel phantom exercises cross-sections and its consequences for the calculated ionization cluster
size distributions (Villagrasa et al., 2022). In the second phase of the
Of the voxel-phantom exercises, two involved exposure to an exercise (in preparation), the focus will be on a comparison of the
external point source, emitting either 60Co gamma photons or 10 keV cross-section datasets for low-energy electron transport and a consid­
neutrons respectively (Huet et al., 2022). The task to be solved was to eration of their impact on the dispersion of nanodosimetric results.
calculate the organ absorbed doses and the effective dose for a given
exposure duration and activity of the source. 2.4. Nanoparticle exercise
A third voxel-phantom exercise also dealt with point source geom­
etries, but for cases of typical X-ray examinations (Huet et al., 2022). In the nanoparticle exercise, the dose enhancement from a gold
Here the task was more complex, as participants were required to nanoparticle, as well as the energy spectrum of electrons emitted from it,
determine the position of the radiation point source in relation to the were to be determined when irradiated with two low-energy X-ray
phantom. In addition, the results were to be presented as conversion spectra. The geometry was simply a gold sphere in water irradiated with
coefficients to organ absorbed doses, both from air kerma and kerma a parallel beam from a plane photon source, the cross-sectional area of
area product. This exercise was thus linked to a potential practical which was slightly larger than that of the nanoparticle. Two different
application in which the latter quantities are determined as part of the nanoparticle diameters were considered, and the dose enhancement was
quality assurance of radiological equipment, and the conversion co­ to be determined in spherical water shells around the nanoparticle (Li
efficients sought would enable an assessment of the dose absorbed by et al., 2020a, 2020b; Rabus et al., 2021b, 2021c).
the patient during the X-ray examination.
The fourth voxel-phantom exercise considered a uniform planar 2.5. Foetus dose during maternal proton therapy
source of 60 keV photons beneath the phantom (mimicking ground
contamination by 241Am) and required participants to calculate organ This exercise consisted of two parts. The first part dealt with the
absorbed dose rates and the effective dose rate for a given area density of effects of different calculation phantoms for pregnant women, and
the emission rate (Eakins et al., 2021). In the fifth voxel-phantom ex­ different code versions of MCNP, on the predicted dose to the foetus
ercise, a mixed radiation field of gaseous 16N was considered, emitting during maternal brain proton therapy. The second part dealt with the
beta and high energy gamma radiation from both inside (lung) and dependence of the secondary neutron spectra on the Monte Carlo radi­
outside the human body (Gómez-Ros et al., 2021); the ratios of organ ation transport codes and nuclear models that were used, and their ef­
equivalent dose rates to activity concentration were to be determined. fects on the calculated and measured neutron doses during proton
The most extensive voxel-phantom exercise involved an idealized therapy (De Saint-Hubert et al., 2021, 2022).
case of internal dosimetry (Zankl et al., 2021c). For the sake of
simplicity, hypothetical radionuclides were considered that were uni­
formly distributed in specified organs and emitted monoenergetic
photon or electrons. Here, absorbed fractions and specific absorbed
fractions of energy in the “source” organ and in specified “target” organs

2
H. Rabus et al. Radiation Measurements 156 (2022) 106822

3. Experiences from the exercises evident after the first publication of the results (Li et al., 2020a) and
required a thorough re-evaluation (Li et al., 2020b), consistency checks
3.1. Observations on participants’ results provided clear indications of the causes of the discrepancies for some
results. Nevertheless, some of the participants concerned did not provide
In general, the ensemble of participants’ results that were submitted revised solutions (Rabus et al., 2021b). However, it must be also stressed
initially showed a large scatter. In the exercises for which a reference that the majority of participants were very supportive of the re-analysis
solution was available, excellent agreement within the expected statis­ of the results and were eager to clarify the origin of the discrepancies
tical fluctuations was found in some cases, while others showed signif­ found initially.
icantly larger deviations, which in some individual cases ranged by up to
several orders of magnitude. For the tasks without a reference solution, a 3.2. Issues with omitted quality assurance of results
subset of the reported results also agreed with each other to some extent,
while others deviated significantly from this group. In both classes of Many of the anomalies found in the reported data could have been
exercise, the occurrence of extreme outliers was not correlated with the detected by the participants themselves, e.g. through simple plausibility
complexity of the problem. checks of their results. Examples are briefly discussed in the following.
Some of the deviations were attributable to simple errors, such as For example, a very general plausibility consideration is that if the
copy-and-paste mistakes or incorrect arrangement of the results in the irradiation conditions are quite homogeneous, it may be expected that
given template. Others resulted from misunderstanding how the final all organ doses will be of broadly similar magnitudes; a single organ dose
results should be normalised (e.g. normalising to the correct quantity result differing by several orders of magnitude from the rest of a given
but at a different distance from the source than was required). In the participant’s dataset ought therefore to be immediately apparent to
voxel-phantom exercise for the case of X-ray examinations, some par­ them as being potentially erroneous. Similarly, if multiple energies are
ticipants normalised to the value of air kerma free in air at a specific considered, it is unlikely that the value for a single intermediate energy
distance from the source or to the entrance surface dose (which includes will be entirely outside the range of values for all other energies.
backscatter) instead of to air kerma free in air at the skin as was In the Bonner sphere exercise, there were cases of reported results
requested. In the microdosimetric and nanodosimetric in­ with negative values for the neutron fluence. These physically impos­
tercomparisons, the normalisation to ‘one decay of the electron source’ sible values, as well as anomalous spectral shapes, could have been
was not always understood by the participants and was also sometimes identified by simply plotting the results. Some of the reported spectra
difficult to implement for some Monte Carlo codes. The use of a loga­ differed from the reference solutions by several orders of magnitude;
rithmic scale for the microdosimetric quantity (specific energy distri­ such anomalies could have been easily detected if the unfolded spectra
bution) also caused problems with proper normalisation. had been convolved with the given sensitivities of the Bonner spheres, to
Many major deviations were caused by the fact that the participants’ verify that the given count rate was then achieved.
simulations deviated from the specifications in terms of geometrical In the voxel phantom exercise for internal dosimetry, a simple
dimensions or the quantities to be determined. One example of this was plausibility check would have been that the absorbed fraction for elec­
the “nanoparticle” exercise, where only four out of eleven participants trons and low-energy photons must be close to unity in a source organ
implemented the requested geometry correctly, which consisted of a and quite small for other organs, since these radiations have a short
gold sphere irradiated in water by a collimated parallel photon beam of range in condensed matter and therefore deposit their energy close to
given dimensions. Another example was the voxel phantom exercise on the point of release. Moreover, some participants in this exercise re­
internal dosimetry, where some participants used organ masses that ported results for absorbed fractions and specific absorbed fractions for
included blood instead of those given in ICRP Publication 110, as was which the ratio of these two quantities varied between different energies
stated in the exercise definition. In some of the voxel-phantom exercises, of the particles emitted from the (monoenergetic) source. However,
the choice of the location of the source was also sometimes a problem since this ratio is simply the mass of the organ, it cannot depend on the
due to deviation from the correct reference point (e.g. the edge of the energy.
phantom array instead of the phantom’s skin). In addition, for many of the tasks in the voxel phantom exercises,
Other causes of major deviations were that some participants were literature values are available for fairly similar exposure conditions that
not familiar with certain concepts, such as the normalisation quantity could have been used for comparison, at least as a first approximation to
“kerma area product” (in the voxel-phantom X-ray exercise) or effective indicate the expected magnitudes of the results.
dose; mistakes for the latter included not applying tissue weighting When dealing with voxel phantom simulation, one of the simplest
factors correctly, not averaging and summing over the defined set of checks might be to visualise the problem in order to ensure the proper
organs, neglecting to sex-average, or neglecting to apply the correct positioning of the beam, though it is noted that some software packages
energy-dependent radiation weighting factor for neutron exposures. struggle due to the sizes of these input files. If one is using any variance
In the voxel-phantom exercises, many participants had problems reduction it should also be ensured that simulations with and without
applying the method recommended for bone marrow dosimetry in application of these techniques reproduce the same results, albeit with
(ICRP, 2010). This finding stimulated writing an article to better explain differing statistical uncertainties.
this approach, which is also part of this Special Issue (Zankl et al.,
2021a). 3.3. Issues with exercise definitions
As already mentioned, participants whose results differed from the
reference solution (class 1) or from the majority of other participants’ In some cases, inadequacies in exercise definitions became apparent
solutions (class 2) were informed of this fact and asked to revise their while they were already running. In the nanoparticle exercise, for
solutions. Not all contacted participants responded to this invitation or example, one of the quantities to be reported by the participants was the
provided the requested information on details about their simulations. energy spectrum of electrons “in spherical shells” around the nano­
Of those participants who submitted a revised solution, some did not particle, with the radii of the bounding spherical surfaces given.
indicate what they had changed in their computational procedure to Most of the participants interpreted this physically undefined
arrive at their revised results. This therefore does not give any additional quantity as the energy distribution of the electrons entering the
insight into possible similar errors to be expected in future similar ex­ respective volume. However, one participant determined the energy
ercises, or hints that could have been communicated to the other distribution of the balance of the number of electrons traversing the
participants. surfaces of the respective volume and withdrew her results on the
In the nanoparticle exercise, where some inconsistencies became assumption that the observed negative frequencies indicated an error

3
H. Rabus et al. Radiation Measurements 156 (2022) 106822

that she could not locate. daily work. Therefore, the topics of the intercomparison exercises must
Another problem with this part of the nanoparticle exercise was that be relevant to the participants’ fields of work, and the tasks to be solved
there was no default energy binning, so participants chose very different should not be overly demanding in terms of setup time. CPU resource
values for the bin size, with some using logarithmic binning and others requirements may also need to be considered but should generally be
using linear binning. The large statistical variations in the results ob­ less of an issue.
tained with small energy bin sizes masked the variations between the To meet the workload requirements, some of the exercises presented
different results when plotted together (Li et al., 2020a). in this Special Issue were designed with simplified idealistic geometric
In the exercise on Bonner sphere spectrum unfolding, it was found setups and irradiation conditions. Examples include the voxel phantom
during the analysis that in one of the scenarios considered, there was an exercises for monoenergetic point sources, the uncertainty exercise in
interference of the Bonner sphere response due to backscattering of micro- and nanodosimetry (idealized energy spectrum), and the nano­
neutrons from a nearby concrete wall (Gómez-Ros et al., 2018). particle exercise (simplistic geometry). These simplifications have
In the voxel phantom exercises featuring 60Co photons and 10 keV sometimes raised concerns among reviewers about the usefulness of the
neutrons, the instruction given to participants for the location of the respective comparisons but seem justified given the aforementioned
point source was to place it ‘100 cm from the surface of the chest’, which time constraints.
could be interpreted differently. In response, the organizers performed Regardless of the complexity of a task, a complete description of the
small sensitivity analyses to quantify the impact from the ambiguity of problem to be solved with all relevant information must always be
this parameter, the outcomes from which were used to imply appro­ given. For Monte Carlo simulation exercises, this means that the radia­
priate ‘tolerances’ that could be applied to the submitted results (Huet tion source, simulation geometry and materials must be comprehen­
et al., 2022). sively specified. On the other hand, it should generally not be specified
The voxel phantom exercise on internal dosimetry was not wisely exactly how the Monte Carlo simulation or the unfolding are to be
designed in several respects. The tasks to be solved were too extensive, carried out. The path to the solution, as well as the tools to be used, must
which also made evaluation challenging and led to delays in feedback to be decided at the discretion of the participant. The participant must
the participants. For electrons and low-energy photons, the source and determine, for example, whether and which variance reduction tech­
target organs were sometimes too far apart, which led to very large niques can or should be used, whether the transport of secondary
statistical uncertainties even in the reference solution. Therefore, the charged particles should be simulated or how the thermal neutron
degree of deviation between participant and master solutions could not transport should be performed.
be reliably quantified in some situations. However, depending on the aim of the exercise, a more detailed
In the uncertainty exercise, the use of a multi-energy electron source specification of intermediate steps or procedures may be advisable. For
that was similar to the 125I decay but did not take into account the instance, whenever the performance of codes or their differences is to be
variability of the actual decay, complicated the understanding of the assessed, it may be wise also to specify some of the aforementioned
problem on one hand and, on the other hand, did not favour the analysis aspects of the simulations to ensure that the differences between results
of the sensitivity study on the variation of the cross sections. Indeed, the from different participants only reflect the differences in the codes that
use of monoenergetic electrons would have helped in both aspects. one is interested in.
In the foetus dose exercise, atomic numbers of the elements, mass
numbers of the nuclides, and cross-section identifiers had not been fixed 4.2. Reporting of results
for all materials used in the simulations, so participants made their own
(different) choices, which caused some of the discrepancies initially The task definition should include very precise instructions for
noted. reporting results, and templates should be provided where possible.
Providing such a template, where the participants were requested to fill
3.4. Issues with the timetable of the exercises in their results in a pre-defined format, not only helps clarify exactly
what output is required from them in each case, but also greatly facili­
Most exercises were planned with a timetable, which in almost all tated the evaluation of the results in the respective exercises. This is
cases proved to be too ambitious and optimistic. This was partly because especially important when spectral information is to be reported, where
for many exercises the initial number of participants was lower than was a lack of specification of bin division can make synopsis quite cumber­
expected and considered adequate for the purpose, so submission some when different participants use different bin sizes and/or linear
deadlines were postponed several times to increase participation after and logarithmic equidistant bins.
further publicity for the exercises. Further deadline extensions became In addition, asking for redundant information can help to identify
necessary at the request of the participants. potential problems with participants’ data. Examples of this were: the
After an initial analysis of the submitted solutions, participants voxel phantom exercise for internal dosimetry, where absorbed fractions
whose results differed by more than expected from either the reference and specific absorbed fractions were to be reported (differing by only
solution or from most other participants’ solutions (as appropriate) were one factor, i.e. organ mass); or the voxel phantom exercise for the X-ray
informed on this fact and invited to revise their solutions. Deadlines for examinations, where the results were to be reported normalised to both
the submission of revised results also had to be postponed several times. kerma free-in-air and kerma area product, which again differ by only
As a result, the total duration of the exercises exceeded the typical one factor. In the case of the nanoparticle exercise, only reporting of
length of stay of junior researchers at a given institute, making it diffi­ results normalised to the number of primary particles was required. If
cult, if not impossible, to follow-up on abnormal results in some cases. normalisation to the area density of the emitted primary photons from
the source had also been reported, the incorrect implementations of the
4. Lessons learnt simulation geometry would have been detected much earlier during the
exercise.
This section discusses the insights gained from the exercises from the
perspective of the organizers. 4.3. Timing of the exercise

4.1. Problem specification Regarding the problems encountered with non-responding partici­
pants at the revision of results stage, it is planned to set up rules in future
The participants, as well as the organisers of the exercises, are exercises to get a more formal commitment from participants. The rules
committed to EURADOS and the intercomparisons in addition to their to be established concern deadlines, participation in the feedback loop,

4
H. Rabus et al. Radiation Measurements 156 (2022) 106822

and requirements for co-authorship to potential manuscripts (see Sup­ given more emphasis in the future. This is, of course, the responsibility
plementary Fig. S1 and Supplementary Fig. S2). of each participant, but appropriate commitments can be included in the
In addition, more timely feedback to the participants might improve application forms (Supplementary Fig. S1 and Supplementary Fig. S2).
their preparedness to disclose details of their computational procedures As a minimum requirement, participants should provide documentation
and improvements. Long feedback intermissions make it difficult for the on where the following files are stored and backed up:
participants to recall exactly what was done and even what the exercise
was about. It should be kept in mind that the organizers, as well as all – data files containing the reported results
participants, are performing these exercises alongside their daily duties. – data files with the raw simulation output
In the reanalysis of the nanoparticle exercise, a set of hierarchical MS – files used for their production (material data or other input files,
Excel templates were used that allowed a fast assessment of the internal code, etc.)
consistency of the results reported by participants (in an Excel-template – log files and other supplementary output files of the simulations.
provided to them) as well as a ‘live’ synopsis via hyperlinks. As an
illustration of this approach, Supplementary Fig. S3 to Supplementary This information is essential when participants need to review their
Fig. S6 show screenshots of the “Synopsis” worksheets with easy-to- work for possible errors. However, in the exercises, delayed responses
assess graphs and calculated figures of merit (integral quantities nor­ from participants were sometimes explained by difficulties in finding
malised so that their expected values are close to unity). data, uncertainty about which version of a code was used, and similar
Using these templates to assess the consistency of a participant’s such problems. Therefore, it may be useful for the organisers to collect
results took only a few minutes and required only copying and pasting this information - or even the files containing the metadata of the sim­
the results from the Excel templates completed by the participant into ulations - as part of the reporting of the results. This might be the case
the template used for the analysis. In many cases, this enabled feedback especially for participants who are inexperienced users of simulation
in less than an hour. Of course, it was more time-consuming to identify codes, who may not be aware that the log files etc. generated by their
the more sophisticated deviations from the exercise specifications. codes contain information that complements their simulation results
Another issue with timing is that calculations with voxel phantoms and is important for their quality.
may require large amounts of time. With some codes, even the visual­ It was not uncommon for files of original simulation results, which
isation of them can take up to several hours, and the production calcu­ were shared by participants with the organisers during the feedback
lations even weeks. Some codes have ability to skip the geometry check loops, to contain only columns of numbers. A header indicating which
at the beginning of production calculations (e.g. DBCN card in MCNP) quantities are listed and which units have been used, and ideally also
and in this way speed up the calculations significantly. containing information on the code used and its version, as well as the
date when the file was written, would be minimum requirements for the
4.4. Quality assurance of results useability of these data files. In addition, the aforementioned auxiliary
information is also needed.
As indicated in Section 3.2, in all exercises some of the participants
seemed to have submitted their results without first carrying out 5. Conclusions
adequate quality control of their solutions, e.g. by simple plausibility
checks or by comparison with literature data, if available. Approaches Beyond doubt, the reported EURADOS exercises are beneficial to the
such as the one mentioned in Section 4.2 could allow for faster identi­ field of computational dosimetry. They directly contribute to the
fication of outliers and more timely feedback to participants. This could training of the participants by improving their computational proced­
alleviate some of the problems, such as where there was a lack of ures through feedback with the task organisers. They lead also to the
response from participants following feedback regarding abnormal availability of representative dose values for various exposure condi­
results. tions that may aid future novice users in the quality assurance of their
However, a significant degradation in the quality of results may methods. In addition, they also provide a snapshot of how well (or
persist, as the evolution of many computational tools towards greater otherwise) the computational techniques are being applied within the
ease of use also allows their use without a certain level of expertise, community in general, and how well some of the concepts recommended
which may still be required for meaningful results. Interaction with by organizations such as ICRP are understood; the observed difficulties
some exercise participants who reported unreasonable results revealed a in correctly defining and evaluating effective doses (Eakins et al., 2021;
lack of understanding of several fundamental aspects. For example, that Huet et al., 2022), or in determining bone marrow doses (Zankl et al.,
the results of calculations are not just numbers, but physical quantities 2021a), are clear examples of the latter.
(which have dimensions). The obvious question of what could be done better in future exercises
Some participants were also unaware that there are different ways of has been partly addressed in Section 4. A general answer to this question
specifying the categorical variable of a histogram (lower or upper limit is not easy, since it depends on the objective of the exercises. The
of the bin or the bin centre), which can vary between different codes and question will therefore continue to be the subject of discussion within
affect the comparison of results, such as when reported with different EURADOS Working Group 6 when new exercises are prepared.
definitions of the meaning of the values on the x-axis. This could be To avoid participants wasting their time on the tasks of an exercise in
countered by requiring both the lower and upper bin limits to be cases where they are prone to give incorrect results, due to a wrong idea
reported. of the task or ignorance of the dosimetric quantities to be determined,
There were also cases where participants determined ratios with a several modifications of the exercises can be considered. One could be to
finer bin size than specified in the task, and then re-binned their results define explicitly the dosimetric quantities and normalisation quantities
for reporting by averaging the ratios over the larger bins instead of to be used. Another possibility would be to include in the definition of
calculating the ratio between the sum of the numerators and the sum of the task a list of checks to be made by the participants on the results, or
the denominators. Detection of such elementary mistakes requires ac­ even provide them with templates like those used in the reanalysis of the
cess to the original results, so their origins were not always immediately nanoparticle exercise.
apparent to the organizers. Such changes to the exercises would mitigate the risk of potential
Considering that most exercises will lead to publications in the form errors by the participants. However, while this closer guidance may lead
of EURADOS reports or journal articles, compliance with the principles to improved results, this better agreement will only reflect the partici­
of FAIR data (an acronym for findability, accessibility, interoperability pants’ capability to follow detailed instructions and not their actual state
and reusability (Wilkinson et al., 2016),) is also a matter that should be of expertise or their performance in real-world applications. Moreover,

5
H. Rabus et al. Radiation Measurements 156 (2022) 106822

concepts like effective dose are widely used and already well-defined linear accelerator. Ann. Dell Ist. Super. Sanita 53, 314–321. https://doi.org/
10.4415/ANN_17_04_07.
elsewhere; arguably, it should not therefore be the role of EURADOS
De Saint-Hubert, M., Farah, J., Klodowska, M., Romero-Expósito, M.T., Tyminska, K.,
WG6, which focusses on computational dosimetry, to coach radiation Mares, V., Olko, P., Stolarczyk, L., Trinkl, S., 2022. The influence of nuclear models
professionals on the basic concepts that underpin radiological and Monte Carlo radiation transport codes on stray neutron dose estimations in
protection. proton therapy. Radiat. Meas. 150, 106693 https://doi.org/10.1016/j.
radmeas.2021.106693.
Most of the exercises were the types of task that the participants may De Saint-Hubert, M., Tymińska, K., Stolarczyk, L., Brkić, H., 2021. Fetus dose calculation
be confronted with in their professional activities or in research. during proton therapy of pregnant phantoms using MCNPX and MCNP6.2 codes.
Generally, they then would have to perform the simulation or unfolding, Radiat. Meas. 149, 106665 https://doi.org/10.1016/j.radmeas.2021.106665.
Eakins, J., Huet, C., Brkić, H., Capello, K., Desorgher, L., Epstein, L., Hunt, J.G., Kim, H.
and calculate the dosimetric quantities of interest, without specific S., Krstic, D., Lee, Y.-K., Manohari, M., Nikezic, D., Shukrun, R.H., Souza-Santos, D.,
guidance. A supervisor of an early-stage researcher, or a reviewer of Tymińska, K., 2021. Monte Carlo calculation of organ and effective dose rates from
potential papers arising from such work, can be a source of feedback that ground contaminated by Am-241: results of an international intercomparison
exercise. Radiat. Meas. 148, 106649 https://doi.org/10.1016/j.
hopefully reveals major non-plausibilities in the results, if any. How­ radmeas.2021.106649.
ever, as discussed by (Rabus et al., 2021c), these potential quality filters Gómez-Ros, J.M., Bedogni, R., Domingo, C., Eakins, J.S., Roberts, N., Tanner, R.J., 2022.
often seem to have failed for simulation studies on nanoparticles so far. Results of the EURADOS international comparison exercise on neutron spectra
unfolding in Bonner spheres spectrometry. Radiat. Meas. 153, 106755 https://doi.
A better option could therefore be to create a questionnaire to assess org/10.1016/j.radmeas.2022.106755.
the knowledge of the participants, and then give more detailed in­ Gómez-Ros, J.M., Bedogni, R., Domingo, C., Eakins, J.S., Roberts, N., Tanner, R.J., 2018.
structions to the less experienced participants. One could also start the International comparison exercise on neutron spectra unfolding in bonner spheres
spectrometry: problem description and preliminary analysis. Radiat. Protect. Dosim.
exercise with a webinar explaining what is expected, what participants
180, 70–74. https://doi.org/10.1093/rpd/ncy002.
should do, and explaining the importance of quality assurance. A Gómez-Ros, J.M., Moraleda, M., Arce, P., Bui, D.-K., Dang, T.-M.-L., Desorgher, L.,
recording of the webinar could also be made available on the EURADOS Kim, H.S., Krstic, D., Kuć, M., Le, N.-T., Lee, Y.-K., Nguyen, N.-Q., Nikezic, D.,
website so that participants who join the exercise later can refer to it. Tymińska, K., Vrba, T., 2021. Monte Carlo calculation of the organ equivalent dose
and effective dose due to immersion in a 16N beta source in air using the ICRP
Another potential improvement for some of the exercises, if reference phantoms. Radiat. Meas. 145, 106612 https://doi.org/10.1016/j.
repeated, could be to include several reporting steps in the exercise. For radmeas.2021.106612.
example, in the voxel phantom exercise for the X-ray examinations, Gualdrini, G., Agosteo, S., Ménard, S., Price, R.A., Chartier, J.-L., Großwendt, B.,
Kodeli, I., Leuthold, G.P., Siebert, B.R.L., Tagziria, H., Tanner, R.J., Terrissol, M.,
participants could first need to report their results for the position of the Zankl, M., 2005. QUADOS intercomparison: a summary of photon and charged
radiation source in relation to the phantom; they could then get feed­ particle problems. Radiat. Protect. Dosim. 115, 587–599. https://doi.org/10.1093/
back if it is outside the uncertainty band of the reference value and be rpd/nci197.
Gualdrini, G., Tanner, R.J., Agosteo, S., Pola, A., Bedogni, R., Ferrari, P., Lacoste, V.,
invited to report a revised value. On request they could also get the Bordy, J.-M., Chartier, J.-L., de Carlan, L., Gomez Ros, J.-M., Großwendt, B.,
correct positions as feedback and run their simulations for these. Kodeli, I., Price, R.A., Rollet, S., Schultz, F., Siebert, B., Terrissol, M., Zankl, M.,
Alternatively, such future exercises could have a first step with as few 2008. Analysis of the CONRAD computational problems expressing only stochastic
uncertainties: neutrons and protons. Radiat. Protect. Dosim. 131, 7–14. https://doi.
specifications as possible, a second with plausibility checks to be per­ org/10.1093/rpd/ncn241.
formed by the participants, and a third with feedback on the submitted Harrison, R.M., Ainsbury, E., Alves, J., Bottollier-Depois, J.-F., Breustedt, B.,
results and a request for revision. Caresana, M., Clairand, I., Fantuzzi, E., Fattibene, P., Gilvin, P., Hupe, O.,
Knežević, Ž., Lopez, M.A., Olko, P., Olšovcová, V., Rabus, H., Rühm, W., Silari, M.,
Stolarczyk, L., Tanner, R., Vanhavere, F., Vargas, A., Woda, C., 2021. Eurados
strategic research agenda 2020: vision for the dosimetry of ionising radiation.
Declaration of competing interest
Radiat. Protect. Dosim. 194, 42–56. https://doi.org/10.1093/rpd/ncab063.
Huet, C., Eakins, J., Zankl, M., Gómez-Ros, J.M., Jansen, J., Moraleda, M., Struelens, L.,
The authors declare that they have no known competing financial Akar, D.K., Borbinha, J., Brkić, H., Bui, D.K., Capello, K., Linh Dang, T.M.,
Desorgher, L., Di Maria, S., Epstein, L., Faj, D., Fantinova, K., Ferrari, P., Gossio, S.,
interests or personal relationships that could have appeared to influence
Hunt, J., Jovanovic, Z., Kim, H.S., Krstic, D., Le, N.T., Lee, Y.-K., Murugan, M.,
the work reported in this paper. Nadar, M.Y., Nguyen, N.-Q., Nikezic, D., Patni, H.K., Santos, D.S., Tremblay, M.,
Trivino, S., Tymińska, K., 2022. Monte Carlo calculation of organ and effective doses
due to photon and neutron point sources and typical X-ray examinations: results of
Acknowledgements an international intercomparison exercise. Radiat. Meas. 150, 106695 https://doi.
org/10.1016/j.radmeas.2021.106695.
The support of all members of EURADOS WG 6 who contributed to ICRP, 2009. ICRP publication 110: adult reference computational phantoms. Ann. ICRP
39, 1—164. https://doi.org/10.1016/j.icrp.2009.09.001.
the organization of the exercises and the efforts of the participants to
ICRP, N., 2010. ICRP Publication 116. Conversion coefficients for radiological protection
solve the tasks are gratefully acknowledged. The authors also thank the quantities for external radiation exposures. Ann. ICRP 40, 1–257. https://doi.org/
Editor-in-Chief of Radiation Measurements, Adrie de Bos, for making 10.1016/j.icrp.2011.10.001.
this special issue possible, and the EURADOS Council for providing the Li, W.B., Belchior, A., Beuve, M., Chen, Y.Z., Maria, S.D., Friedland, W., Gervais, B.,
Heide, B., Hocine, N., Ipatov, A., Klapproth, A.P., Li, C.Y., Li, J.L., Multhoff, G.,
necessary funding. Poignant, F., Qiu, R., Rabus, H., Rudek, B., Schuemann, J., Stangl, S., Testa, E.,
Villagrasa, C., Xie, W.Z., Zhang, Y.B., 2020a. Intercomparison of dose enhancement
ratio and secondary electron spectra for gold nanoparticles irradiated by X-rays
Appendix A. Supplementary data
calculated using multiple Monte Carlo simulation codes. Phys. Med. 69, 147–163.
https://doi.org/10.1016/j.ejmp.2019.12.011.
Supplementary data to this article can be found online at https://doi. Li, W.B., Beuve, M., Maria, S.D., Friedland, W., Heide, B., Klapproth, A.P., Li, C.Y.,
Poignant, F., Rabus, H., Rudek, B., Schuemann, J., Villagrasa, C., 2020b.
org/10.1016/j.radmeas.2022.106822.
Corrigendum to “Intercomparison of dose enhancement ratio and secondary electron
spectra for gold nanoparticles irradiated by X-rays calculated using multiple Monte
References Carlo simulation codes” [Phys. Med. 69. Phys. Med. 80, 383–388. https://doi.org/
10.1016/j.ejmp.2020.10.008 (2020) 147-163].
Price, R.A., Gualdrini, G., Agosteo, S., Ménard, S., Chartier, J.-L., Großwendt, B.,
Broggio, D., Bento, J., Caldeira, M., Cardenas-Mendez, E., Farah, J., Fonseca, T.,
Kodeli, I., Leuthold, G.P., Siebert, B.R.L., Tagziria, H., Tanner, R.J., Terrissol, M.,
Konvalinka, C., Liu, L., Perez, B., Capello, K., Cowan, P., Cruzate, J.-A., Freire, L.,
Zankl, M., 2006. Pitfalls and modelling inconsistencies in computational radiation
Gómez-Ros, J.-M., Gossio, S., Heide, B., Huikari, J., Hunt, J., Kinase, S., Kramer, G.
dosimetry: lessons learnt from the QUADOS intercomparison. Part II: photons,
H., Kurihara, O., Kyrieleis, A., Lebacq, A.-L., Leone, D., Li, C., Li, J., Mihailescu, L.-C.,
electrons and protons. Radiat. Protect. Dosim. 118, 155–166. https://doi.org/
Moraleda, M., Navarro, J.-F., Oliveira, C., Puerta, N., Reichelt, U., Simões, C.,
10.1093/rpd/ncl014.
Sommer, D., Takahashi, M., Teles, P., Vanhavere, F., Vrba, T., Franck, D.,
Rabus, H., Gómez-Ros, J.M., Villagrasa, C., Eakins, J., Vrba, T., Blideanu, V., Zankl, M.,
Gualdrini, G., Lopez, M.-A., 2012. Monte Carlo modelling for the in vivo lung
Tanner, R., Struelens, L., Brkić, H., Domingo, C., Baiocco, G., Caccia, B., Huet, C.,
monitoring of enriched uranium: results of an international comparison. Radiat.
Ferrari, P., 2021a. Quality assurance for the use of computational methods in
Meas. 47, 492–500. https://doi.org/10.1016/j.radmeas.2012.04.020.
dosimetry: activities of EURADOS Working Group 6 ’Computational Dosimetry.
Caccia, B., Le Roy, M., Blideanu, V., Andenna, C., Arun, C., Czarnecki, D., El Bardouni, T.,
J. Radiol. Prot. 41, 46–58. https://doi.org/10.1088/1361-6498/abd914.
Gschwind, R., Huot, N., Martin, E., Zink, K., Zoubair, M., Price, R., De Carlan, L.,
2017. EURADOS intercomparison exercise on Monte Carlo modelling of a medical

6
H. Rabus et al. Radiation Measurements 156 (2022) 106822

Rabus, H., Li, W.B., Nettelbeck, H., Schuemann, J., Villagrasa, C., Beuve, M., Di Maria, S., results. Radiat. Meas. 150, 106675 https://doi.org/10.1016/j.
Heide, B., Klapproth, A.P., Poignant, F., Qiu, R., Rudek, B., 2021b. Consistency radmeas.2021.106675.
checks of results from a Monte Carlo code intercomparison for emitted electron Vrba, T., Broggio, D., Caldeira, M., Capello, K., Fantínová, K., Franck, D., Gómez-Ros, J.
spectra and energy deposition around a single gold nanoparticle irradiated by X-rays. M., Hunt, J., Kinase, S., Leone, D., Lombardo, P.A., Manohari, M., Marzocchi, O.,
Radiat. Meas. 147, 106637 https://doi.org/10.1016/j.radmeas.2021.106637. Moraleda, M., Nogueira, P., Ośko, J., Arron, S., Suhl, S., Takahashi, M., Teles, P.,
Rabus, H., Li, W.B., Villagrasa, C., Schuemann, J., Hepperle, P.A., de la Rosales, L.F., Tremblay, M., Tymińska, K., Lopez, M.A., Tanner, R., 2015. EURADOS
Beuve, M., Maria, S.D., Klapproth, A.P., Li, C.Y., Poignant, F., Rudek, B., intercomparison exercise on MC modelling for the in-vivo monitoring of AM-241 in
Nettelbeck, H., 2021c. Intercomparison of Monte Carlo calculated dose enhancement skull phantoms (Part II and III). Radiat. Phys. Chem. 113, 59–71. https://doi.org/
ratios for gold nanoparticles irradiated by X-rays: assessing the uncertainty and 10.1016/j.radphyschem.2015.04.009.
correct methodology for extended beams. Phys. Med. 84, 241–253. https://doi.org/ Vrba, T., Nogueira, P., Broggio, D., Caldeira, M., Capello, K., Fantínová, K., Figueira, C.,
10.1016/j.ejmp.2021.03.005. Hunt, J., Leone, D., Murugan, M., Marzocchi, O., Moraleda, M., Shutt, A., Suh, S.,
Rühm, W., Ainsbury, E., Breustedt, B., Caresana, M., Gilvin, P., Knežević, Ž., Rabus, H., Takahashi, M., Tymińska, K., Lopez, M.A., Tanner, R., 2014. EURADOS
Stolarczyk, L., Vargas, A., Bottollier-Depois, J.F., Harrison, R., Lopez, M.A., intercomparison exercise on MC modeling for the in-vivo monitoring of Am-241 in
Stadtmann, H., Tanner, R., Vanhavere, F., Woda, C., Clairand, I., Fantuzzi, E., skull phantoms (Part I). Radiat. Phys. Chem. 104, 332–338. https://doi.org/
Fattibene, P., Hupe, O., Olko, P., Olšovcová, V., Schuhmacher, H., Alves, J.G., 10.1016/j.radphyschem.2013.12.010.
Miljanic, S., 2020. The European radiation dosimetry group – review of recent Wilkinson, M.D., Dumontier, M., Aalbersberg, IjJ., Appleton, G., Axton, M., Baak, A.,
scientific achievements. Radiat. Phys. Chem. 168, 108514. https://doi.org/10.101 Blomberg, N., Boiten, J.-W., da Silva Santos, L.B., Bourne, P.E., Bouwman, J.,
6/j.radphyschem.2019.108514. Brookes, A.J., Clark, T., Crosas, M., Dillo, I., Dumon, O., Edmunds, S., Evelo, C.T.,
Rühm, W., Bottollier-Depois, J.F., Gilvin, P., Harrison, R., Knežević, Ž., Lopez, M.A., Finkers, R., Gonzalez-Beltran, A., Gray, A.J.G., Groth, P., Goble, C., Grethe, J.S.,
Tanner, R., Vargas, A., Woda, C., 2018. The work programme of EURADOS on Heringa, J., ’t Hoen, P.A.C., Hooft, R., Kuhn, T., Kok, R., Kok, J., Lusher, S.J.,
internal and external dosimetry. Ann. ICRP 47, 20–34. https://doi.org/10.1177/ Martone, M.E., Mons, A., Packer, A.L., Persson, B., Rocca-Serra, P., Roos, M., van
0146645318756224. Schaik, R., Sansone, S.-A., Schultes, E., Sengstag, T., Slater, T., Strawn, G., Swertz, M.
Siebert, B.R.L., Tanner, R.J., Chartier, J.-L., Agosteo, S., Großwendt, B., Gualdrini, G., A., Thompson, M., van der Lei, J., van Mulligen, E., Velterop, J., Waagmeester, A.,
Ménard, S., Kodeli, I., Leuthold, G.P., Price, R.A., Tagziria, H., Terrissol, M., Wittenburg, P., Wolstencroft, K., Zhao, J., Mons, B., 2016. The FAIR Guiding
Zankl, M., 2006. Pitfalls and modelling inconsistencies in computational radiation Principles for scientific data management and stewardship. Sci. Data 3, 160018.
dosimetry: lessons learnt from the QUADOS intercomparison. Part I: neutrons and https://doi.org/10.1038/sdata.2016.18.
uncertainties. Radiat. Protect. Dosim. 118, 144–154. https://doi.org/10.1093/rpd/ Zankl, M., Eakins, J., Gómez Ros, J.-M., Huet, C., 2021a. The ICRP recommended
ncl013. methods of red bone marrow dosimetry. Radiat. Meas. 146, 106611 https://doi.org/
Tanner, R.J., Chartier, J.-L., Siebert, B.R.L., Agosteo, S., Großwendt, B., Gualdrini, G., 10.1016/j.radmeas.2021.106611.
Kodeli, I., Leuthold, G.P., Ménard, S., Price, R.A., Tagziria, H., Terrissol, M., Zankl, M., Eakins, J., Gómez Ros, J.-M., Huet, C., Jansen, J., Moraleda, M., Reichelt, U.,
Zankl, M., 2004. Intercomparison on the usage of computational codes in radiation Struelens, L., Vrba, T., 2021b. EURADOS intercomparison on the usage of the ICRP/
dosimetry. Radiat. Protect. Dosim. 110, 769–780. https://doi.org/10.1093/rpd/ ICRU adult reference computational phantoms. Radiat. Meas. 145, 106596 https://
nch228. doi.org/10.1016/j.radmeas.2021.106596.
Villagrasa, C., Bordage, M.-C., Bueno, M., Bug, M., Chiriotti, S., Gargioni, E., Heide, B., Zankl, M., Gómez Ros, J.-M., Moraleda, M., Reichelt, U., Akar, D.K., Borbinha, J.,
Nettelbeck, H., Parisi, A., Rabus, H., 2019. Assessing the contribution of cross- Desorgher, L., Di Maria, S., El Bakkali, J., Fantinova, K., Ferrari, P., Gossio, S.,
sections to the uncertainty of Monte Carlo calculations in micro- and nanodosimetry. Hunt, J., Jovanovic, Z., Kim, H.S., Krstic, D., Lee, Y.-K., Nadar, M.Y., Nikezic, D.,
Radiat. Protect. Dosim. 183, 11–16. https://doi.org/10.1093/rpd/ncy240. Patni, H.K., Murugan, M., Triviño, S., 2021c. Monte Carlo calculation of organ dose
Villagrasa, C., Rabus, H., Baiocco, G., Perrot, Y., Parisi, A., Struelens, L., Qiu, R., coefficients for internal dosimetry: results of an international intercomparison
Beuve, M., Poignant, F., Pietrzak, M., Nettelbeck, H., 2022. Intercomparison of exercise. Radiat. Meas. 148, 106661 https://doi.org/10.1016/j.
micro- and nanodosimetry Monte Carlo simulations: an approach to assess the radmeas.2021.106661.
influence of different cross-sections for low-energy electrons on the dispersion of