Journal of Pediatric Surgery 58 (2023) 1921e1928

Contents lists available at ScienceDirect

Journal of Pediatric Surgery

journal homepage: www.sciencedirect.com/journal/
journal-of-pediatric-surgery

A Quality Assessment of the ARM-Net Registry Design and

Data Collection
Isabel C. Hageman a, b, *, Hendrik J.J. van der Steeg a, Ekkehart Jenetzky c, d,
Misel Trajanovska b, e, Sebastian K. King b, e, f, Ivo de Blaauw a, Iris A.L.M. van Rooij g, ARM-
Net Consortium1
a
Department of Surgery-Pediatric Surgery, Radboudumc Amalia Children's Hospital, Nijmegen, the Netherlands
b
Surgical Research, Murdoch Children's Research Institute, Melbourne, Australia
c
Faculty of Health/School of Medicine, Witten/Herdecke University, Witten, Germany
d
Department of Child and Adolescent Psychiatry and Psychotherapy, University Medical Center of the Johannes-Gutenberg-University, Mainz, Germany
e
Department of Paediatrics, University of Melbourne, Melbourne, Australia
f
Department of Paediatric Surgery, The Royal Children's Hospital, Melbourne, Australia
g
Department for Health Evidence, Radboudumc, Nijmegen, the Netherlands

a r t i c l e i n f o a b s t r a c t

Article history: Background: Registries are important in rare disease research. The Anorectal Malformation Network
Received 23 December 2022 (ARM-Net) registry is a well-established European patient registry collecting demographic, clinical, and
Received in revised form functional outcome data. We assessed the quality of this registry through review of the structure, data
27 January 2023
elements, collected data, and user experience.
Accepted 8 February 2023
Material and methods: Design and data elements were assessed for completeness, consistency, useful-
ness, accuracy, validity, and comparability. An intra- and inter-user variability study was conducted
Keywords:
through monitoring and re-registration of patients. User experience was assessed via a questionnaire on
Anorectal malformations
Patient registry
registration, design of registry, and satisfaction.
Quality Results: We evaluated 119 data elements, of which 107 were utilized and comprised 42 string and 65
Rare diseases numeric elements. A minority (37.0%) of the 2278 included records had complete data, though this
improved to 83.5% when follow-up elements were excluded. Intra-observer variability demonstrated
11.7% incongruence, while inter-observer variability was 14.7%. Users were predominantly pediatric
surgeons and typically registered patients within 11e30 min. Users did not experience any significant
difficulties with data entry and were generally satisfied with the registry, but preferred more longitudinal
data and patient-reported outcomes.
Conclusions: The ARM-Net registry presents one of the largest ARM cohorts. Although its collected data
are valuable, they are susceptible to error and user variability. Continuous evaluations are required to
maintain relevant and high-quality data and to achieve long-term sustainability. With the recommen-
dations resulting from this study, we call for rare disease patient registries to take example and aim to
continuously improve their data quality to enhance the small, but impactful, field of rare disease
research.
Level of Evidence: V.
© 2023 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND
license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

Abbreviations: ARM, Anorectal malformations; ARM-Net, Anorectal malforma- 1. Background

tions network; EDC, Electronic data capture; ICD, International classification of
diseases; IQR, Interquartile range; OMIM, Online mendelian inheritance in man;
ORDO, Orphanet rare disease ontology; PROM, Patient-reported outcome measure; Anorectal malformations (ARM) are congenital defects involving
Radboudumc, Radboud University medical center; REDCap, Research electronic the anus and rectum and have an estimated prevalence in Europe of
data capture. 20e33 per 100,000 live births [1,2]. With the rarity of this disease,
* Corresponding author. Radboud University Medical Center, Geert Grooteplein data are scarce and scattered. Therefore, patient registries, as in-
Zuid 10, 6500HB Nijmegen, the Netherlands.
E-mail address: [email protected] (I.C. Hageman).
formation systems that gather clinical data from different health
1
The full members belonging to the ARM-Net consortium not mentioned as care centers, play a pivotal role in rare disease research [3,4].
contributors are written down below in the Acknowledgements section.

https://doi.org/10.1016/j.jpedsurg.2023.02.049
0022-3468/© 2023 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
1922 I.C. Hageman, H.J.J. van der Steeg, E. Jenetzky et al. / Journal of Pediatric Surgery 58 (2023) 1921e1928

The ARM Network (-Net) Consortium, a group of dedicated user experience. All data were collected in Microsoft Excel 2016
European pediatric surgeons, epidemiologists, geneticists, and pa- (v16.62, Microsoft Corporation, Redmond, Washington, United
tient advocacy groups, was established in 2010 [5]. The ARM-Net States) and, where applicable, analyzed in Stata (v17.0, StataCorp,
Consortium has contributed significantly to the field of ARM, with College Station, Texas, United States).
its consensus and publications to harmonize diagnostic classifica-
tions, interventional approaches, and psychosocial follow-up 2.1. Assessment of registry structure and data elements
[6e8]. Furthermore, it has facilitated ARM patients and their fam-
ilies to provide input and collaborate with clinicians to enhance All 119 data elements in nine categories and accompanying
colorectal care (www.arm-net.eu). value labels were given a reference number (Supplementary File 1).
The ARM-Net Consortium also established the ARM-Net patient For each data element and accompanying data of 2287 records, the
registry, for which pediatric surgeons identified the data elements number of missing data, actions for cleaning, actions for analysis,
to be collected and these were converted and developed into an and suggestions for improvements were recorded as well as
online interface. Since 2010, this project of dedicated volunteers assessment of accuracy, completeness, consistency, usefulness, and
has successfully registered more than 2200 ARM patients across validity [4,18e22]. Completeness analysis of records was conduct-
Europe. The original intention of the registry was to prospectively ed, and completion rate was considered 100% if data was entered
collect anonymized data on all consecutive patients of the partici- for each element. This analysis excluded the automatically created
pating centers to surveil epidemiology of ARM and create a elements (n ¼ 8, as these were never missing), unused elements
screening list of potential subjects for studies in ARM research. (n ¼ 4), completion check elements (n ¼ 2), and non-applicable
However, as the registry developed and grew over the years, it has elements if not performed (n ¼ 17, e.g., stoma bowel section if
also served purpose to collect data and conduct research on health- patient has no stoma). The remaining free text string elements for
related outcomes for this patient group, including longitudinal further specifications (n ¼ 24) were also excluded, as these were
data. Several important studies have been conducted with the not mandatory to complete and therefore cannot be regarded as
collected data to improve clinical care for ARM [5,9e15]. Identifi- missing data if left blank. A total of 64 elements were included in
cation of risk factors for unfavorable outcomes, interpretation the completeness analysis. Completeness analysis was also con-
variation in diagnostics, and different management approaches are ducted for data elements irrespective of records, meaning
amongst the significant contributions resulting from the ARM-Net completion rate was calculated for each data element and regarded
patient registry. as 100% if there were data for 2287 records. Completeness of one-
Although the registry collects valuable information, it has been year follow-up data elements excluded the records of patients
apparent that not all collected data are useful for documentation whose reconstructive surgery was performed less than one year
purposes, nor directly useable for research. Data cleaning and before date of data extraction from the ARM-Net registry for the
supplying of missing information are both necessary. Additionally, current study. Median completion rate for all elements was calcu-
due to the design of the registry, requested data on individual pa- lated, excluding the previously specified elements.
tients was intentionally limited, as not to discourage surgeons to
register their data. As a consequence, potentially relevant infor- 2.2. Intra- and inter-user variability
mation for research purposes is not currently captured in the reg-
istry, leading to disparate pieces of information in different places, A monitoring session to study intra- and inter-user variability
or unavailability all together. Although these existing hurdles in- was conducted on ten patients previously registered by a single
fluence the usability and experience with the registry, both from a pediatric surgeon responsible for the registration of patients at the
clinical and a research perspective, the registry has not undergone Radboud University Medical Center (Radboudumc). The ten pa-
fundamental changes since its initial setup. tients were randomly selected based on varying Krickenbeck type
The registry in its current form is very valuable, but it has [8,23], sex, and age to ensure a sample of patients appropriately
outgrown its original purpose over its lifetime. Therefore, updates reflecting the ARM population. All patients were treated at the
and improvements are warranted for the registry to continue to be Department of Pediatric Surgery at the Radboudumc, with suffi-
of high quality and sustainable for long-term use. For this reason, cient time since reconstructive surgery, so as one-year follow-up
this study aimed to assess the quality of the ARM-Net registry data should be available for both first- and second-time registra-
through analysis of the structure and data elements, collected data, tion. The selected patients were registered by a pediatric surgeon
and user experience. More importantly, registries often exhaust (HJJS) for a second time at least one year after first registration for
their funding on initial setup or engaged investigators may disperse intra-user variability of the database (User 1A vs. User 1B), and by a
over time, resulting in few resources left for continued mainte- second user (junior doctor, ICH), for evaluation of differences be-
nance and delivery of high-quality data [4,16]. For this reason, this tween the first and the second user for inter-user variability (User 1
report aimed to highlight the need for other rare disease patient vs. User 2). During second-time registration, User 1 was blinded
registries to continuously self-evaluate and adapt to remain rele- from their first registration in the ARM-Net registry. For data
vant, both for clinical and research purposes. collection, a copy of the ARM-Net registry was built in Research
Electronic Data Capture (REDCap; v12.5.16, Vanderbilt University,
2. Methods Nashville, Tennessee, United States), so as not to affect the existing
ARM-Net registry. Certain data elements, such as elements with
This quality assessment is based on published methods for un- multiple answers, were coded and exported differently, resulting in
dertaking quality evaluation of rare disease registries, including a total number 148 data elements per user per patient. Of these 148
assessment of data with the dimensions of quality, monitoring, elements, five were automatically created by the system and 27
intra- and inter-user variability studies, requesting and providing data elements were free text string elements, resulting in 116 data
feedback and recommendations, and producing a quality evalua- elements included in the analysis. Closed-ended data elements
tion report [4,17e22]. Therefore, the ARM-Net registry quality were considered as either congruent or incongruent. Open-ended
assessment consisted of three parts: critical analysis of the registry free text data elements were analyzed for differences in interpre-
structure and data elements; monitoring of collected data for user tation, which were categorized per subject. The absolute total of
variability; and an ARM-Net registry user questionnaire to evaluate discrepancies in the collected data and the number of data
 I.C. Hageman, H.J.J. van der Steeg, E. Jenetzky et al. / Journal of Pediatric Surgery 58 (2023) 1921e1928 1923

elements with discrepancies were reported. Differences between for patients with a reconstruction more than one year before data
intra- and inter-user variations were tested for significance using extraction.
independent Students’ t-tests and considered statistically signifi- Completeness of data elements does not necessarily equal useful
cant at a p-value of <0.05. data. For many of the elements, there is the option to select ‘Un-
known’. For the data elements on family history of congenital
2.3. ARM-net registry user questionnaire malformations in the parents of the patient, 34.4% and 32.3% of
records had ‘Unknown’ answers for father and mother, respectively.
In collaboration with pediatric surgeons (SKK, IdB, HJJS), ARM The remaining data elements on family history of siblings and
researchers (MT, IALMvR), and ARM-Net data managers (IALMvR, extended family members had an even higher proportion, with a
EJ), the ARM-Net registry user questionnaire was developed and median of 72.8% (IQR 72.4e84.5%) of records ‘Unknown’. This raises
aimed at all ARM-Net users who contribute patients to the registry. the question of relevancy of these elements when information on
It consisted of items on current and future purposes of the registry, extended family members is seemingly often unavailable.
maintenance, satisfaction with collected elements and desired There were 27/107 (25.2%) elements that were free text fields,
changes, ease of use, limitations, and general satisfaction (Supple- allowing a user to add an additional description of a previously
mentary File 2). The questionnaire was built in Castor EDC chosen answer, or when the option ‘Other’ was selected. For these
(v2022.3.0.0, Castor, Amsterdam, The Netherlands) and sent out to data elements, there were 61e577 different answers entered per
all 32 users responsible for data input at their respective centers. item. From the collected data it was evident that the free text ele-
Users were given a minimum period of two months, with three ments are utilized more often than the option ‘Other’ is selected,
reminders, to complete the questionnaire. Percentages of response presumably because the existing answer options in the corre-
are presented without decimals, as the total absolute number of sponding single-choice question were not sufficient or satisfactory.
users that have completed the questionnaire is less than 100. This was especially interesting for the free text element on Krick-
enbeck classification, which was utilized for 283 records, whilst the
3. Results answer option ‘Other’ for the closed-ended Krickenbeck element
was only selected for 20 records. Another reason free text elements
3.1. Assessment of registry structure and data elements were used was to register information that could not be collected
elsewhere, such as prematurity, birth weight, order of birth, and
There were 119 data elements, of which eight were automati- method of conception.
cally created by the electronic data capturing (EDC) system of the The majority of single-choice data elements on associated
ARM-Net registry. Seven involved time and person creating, anomalies collected data on whether the respective organ system
modifying, and locking records. One was a unique identifier to was ‘Abnormal’ or ‘Normal’, whether it was ‘Not checked’, or
anonymize the data and prevent duplicate entries, generated by whether this information was ‘Unknown’. Although these data ele-
entering the date of birth of the patient and the year of birth of the ments have subsequent free text fields for descriptions, users did not
patient's mother. Four elements included in the data dictionary always utilize them. When information was provided in these fields,
were not present in the registry interface and were therefore not it often required extensive cleaning and human interpretation.
used in data collection. Of the remaining 107 utilizable elements,
there were 42 string and 65 numeric data elements (Supplemen- 3.2. Intra- and inter-user variability
tary Table 1). The data elements could be categorized into several
groups: automatically created administrative, patient de- Both User 1 and 2 registered 10 patients, entering 116 data el-
mographics, ARM diagnosis, family history of congenital abnor- ements per patient, totaling 1160 entered datapoints per user.
malities, genetic testing and availability of biosamples, associated Discrepancies between first- and second-time registration (User 1A
anomalies, surgical procedures and associated complications, one- and User 1B) for a single user demonstrated that there is intra-user
year follow-up data, and completion of the record checks (Table 1). variability, with 11.7% of collected data incongruent. Similarly, the
The majority (63.6%) of the 107 utilized data elements were dedi- number of discrepancies between User 1 and User 2 showed that
cated to associated anomalies and surgery. there is also inter-user variability, with 14.7% of collected data
Complete data was available for only 37.0% of records. However, differing between the users (Table 2). The number of intra-user and
when excluding the one-year follow-up data elements (n ¼ 10) inter-user discrepancies were not statistically different (136 vs. 170;
completion rate improved to 83.5% (Fig. 1A). Irrespective of record p  0.20). The discrepancies were found in 61 and 70 of the 116 data
completeness, median completion rate per data element was 99.6% elements for intra- and inter-user variability, respectively. To
(IQR 99.3e99.6%), with a median completion rate of 58.6% (IQR determine which data elements might be specifically sensitive to
51.1e63.3%) for the one-year follow-up data elements only (Fig. 1B) intra- or inter-user variability, the number of data elements with
three or more discrepancies in the 10 patients registered were
determined, but the difference in intra- and inter-user variation
Table 1 (15.5% vs. 23.3%, respectively), was not statistically significant.
Overview of number of data elements per category. For intra-user variation, elements with discrepancies were
Category Number of data elements found in the categories of, in order of frequency, associated
Administrative 8
anomalies (n ¼ 57), surgery and complications (n ¼ 30), family
Demographics 10 history (n ¼ 17), one-year follow-up (n ¼ 15), genetic testing and
ARM diagnosis 2 biosamples (n ¼ 9), patient demographics (n ¼ 6), and diagnosis
Family history 10 (n ¼ 2). The three data elements with the most discrepancies were
Genetic testing and biosamples 5
“Brain abnormality” (n ¼ 10), “Other gastrointestinal abnormality”
Associated anomalies 49
Surgery and complications 19 (n ¼ 8), and “DNA sample” (n ¼ 6). For the first two elements, the
Completion check 2 discrepancies resulted mostly from ‘Normal’ chosen at first-time
One-year follow-up 12 registration, contrary to ‘Not checked’ chosen at second-time
Not used 4 registration. For “DNA sample”, discrepancies came from missing
Total 119
data.
1924 I.C. Hageman, H.J.J. van der Steeg, E. Jenetzky et al. / Journal of Pediatric Surgery 58 (2023) 1921e1928

Fig. 1. (A): Proportion of records with complete data for all data elements and without follow-up elements. (B): Median completion rate per data element of all data elements and
follow-up data elements only.

Table 2
Intra- and inter-user variation.

User 1A vs. user 1B User 1 vs. user 2 p-valueb

Intra-user variation Inter-user variation

Total real discrepancies (n ¼ 1160a) 136 (11.7%) 170 (14.7%) 0.20

Data elements with discrepancies in 1 record(s) (n ¼ 116) 61 (52.6%) 70 (60.3%) 0.24
Data elements with discrepancies in 3 records (n ¼ 116) 18 (15.5%) 27 (23.3%) 0.20
a
116 data elements per patient, with 10 patients resulting in a total of 1160 datapoints.
b
Independent Students' t-test to test for significant differences between the variations.

The data elements subject to inter-user variability fell, in order purpose of the registry was for clinical research and would like the
of frequency, into the categories of associated anomalies (n ¼ 101), future purpose to shift to improving clinical care (71%).
one-year follow-up (n ¼ 26), surgery and complications (n ¼ 22), The majority of users wanted to collect patient-recorded
genetic testing and biosamples (n ¼ 13), family history (n ¼ 3), outcome measures (PROMs) and to expand the follow-up period,
diagnosis (n ¼ 3), and patient demographics (n ¼ 2). Similar to the currently at 5 years, to at least until adulthood, up to lifelong. The
results of intra-user variation, the three data elements with the current registry data element with most votes for removal from the
most discrepancies found were “Other gastrointestinal abnormal- registry was DNA sample, followed by eye/ear abnormalities, family
ity” (n ¼ 10), “Spinal canal/cord specification” (n ¼ 8), and “DNA history, and facial dysmorphic features. Of all users, 21% selected
sample” (n ¼ 6). The discrepancies for “Other gastrointestinal ab- ‘Other’ for removal of an item, which comprised of removal of no
normality” were mostly because User 1 selected ‘Normal’, whereas data elements (15%), removal of free text option for DNA sample
User 2 entered ‘Not Checked’. “Spinal canal/cord specification” (3%), and removal of the dermatological problems data element
mostly had discrepancies because the data were missing by the first (3%). Users were willing to collect more items, with prematurity
user, and the same was for “DNA sample”. This might be because and birthweight, cardiac abnormality consequences, and expansion
the original ARM-Net registry has ‘Unknown’ as the default answer of diagnostic tests as frequently chosen. Other data elements users
when untouched, contrary to the REDCap version, where there suggested for inclusion were incontinence scales, dilatation regi-
were no default answers and non-selection resulted in missing mens, date of stoma placement, menstrual outflow obstruction,
data. methods of conception, and PROM questionnaires on patient/
parent satisfaction, quality of life, sexual function, urinary function,
3.3. ARM-net registry user questionnaire and incontinence/constipation. The majority of users made use of
the free text fields and supported the need to keep this option to
The ARM-Net registry user questionnaire had a response rate of gather specific information, rather than just using an ‘Other’ option
75% (24/32) with complete information provided by 23/24 re- without the availability of specification in a free text field.
spondents. The results of the questionnaire could be categorized in Patients were mostly registered by their treating pediatric sur-
current and future design of the ARM-Net registry, registration of geon (71%) and registration took approximately 11e30 min (61%)
patients, and satisfaction (Table 3). The initial purpose of the reg- for most users. A patient was registered when the pediatric surgeon
istry was surveillance of all consecutive ARM patients in partici- remembered to do so (61%), and follow-up entry was remembered
pating centers. Over a third (38%) of users have indeed selected this with a manual note in the patient's medical file (35%). Users were
as the current purpose, while only 13% believed this is a future generally satisfied (65%) and found the ARM-Net registry easy to
purpose of the registry. Most users (54%) believed the current use (65%), rating the registry with an overall mean grade of 7.4 on a
 I.C. Hageman, H.J.J. van der Steeg, E. Jenetzky et al. / Journal of Pediatric Surgery 58 (2023) 1921e1928 1925

Table 3
Responses of ARM-Net registry user questionnaire.

scale of 1e10, with 1 being useless and 10 being perfect. Users were specifications of previous data elements to information that cannot
largely not willing to pay a yearly fee for maintenance of the reg- be registered anywhere else in the registry. This suggests that the
istry (65%). Those who would pay entered a fee ranging from 25 to registry has outgrown its initial purpose and that existing data el-
500 euros a year, with 50 euros as the most common answer. ements and answer options are not fully satisfactory. Unsatisfactory
data elements and answer options result in room for in-
4. Discussion terpretations and frequent use of free text fields, making the reg-
istry vulnerable to mistakes and imputation errors. The user
The current study evaluated the quality of the ARM-Net registry experience questionnaire also confirmed this, where the majority
through a critical analysis of the structure and data elements, the of users opted to use free text fields and suggested the addition of
collected data, and the user experience. Most data elements were several data elements currently missing in the registry.
dedicated to collecting information on associated anomalies and According to the ARM-Net registry user questionnaire, most
surgery. A quarter of the data elements were free text fields, users registering patients were pediatric surgeons, but still the
resulting in a very high number of up to nearly 600 different an- registry has apparent tendency for differences in data collection, or
swers submitted for a single field, varying from further intra-user and inter-user variability. Up to 15% of collected data was
1926 I.C. Hageman, H.J.J. van der Steeg, E. Jenetzky et al. / Journal of Pediatric Surgery 58 (2023) 1921e1928

incongruent between users and between different timepoints of monitoring should be repeated across all participating centers in
registration by a single user. This shows that discrepancies in data the different countries, with different pediatric surgeons, and
collection exist not only between different users, but also equally covering a larger patient sample. Nevertheless, the small sample of
for the same user. The most common discrepancies were either due patients with two users from the same center and identical in-
to missing data or interpretation differences. Absence of clear structions on patient registration, does not invalidate the findings
documentation in the patient's medical file can either be inter- of variability in the current form of data collection. In order to
preted as a specific item not checked for or considered as normal. confirm the degree of variability, evaluation of intra- and inter-user
Additionally, irrespective of intra- or inter-user variability, the variability is needed on a larger scale across all participating
discrepancies mostly occur in the category of associated anomalies, centers.
surgery and complications, and one-year follow-up, indicating that A third limitation may be the representativeness of the ARM-Net
the elements in these categories should be evaluated for registry user questionnaire. The users who completed the ques-
improvement to minimize imputation errors or left missing. tionnaire were responsible for data collection in large surgical
Several data elements that fall in these categories were also centers and cumulatively registered more than 80% of all records.
selected by the users in the questionnaire to be removed. Therefore, it is unlikely that they are not representative of ARM-Net
Although implemented after first initiation of the registry due to registry users. Finally, the user questionnaire is not a standardized
wishes from the pediatric surgeons, one-year follow-up data entry survey, but rather tailored specifically to the issues data managers
is still an evident weakness of the registry, with the number of experienced with the ARM-Net registry and reviewed by pediatric
complete records being only 37%, while increasing to over 80% surgeons, ARM researchers, and ARM-Net data managers, which
when excluding follow-up data elements. Nevertheless, most users although not validated, we believe is sufficient for the purpose of
reported that more follow-up data should be collected and this study.
preferred to expand the follow-up period to at least until adult- Despite the lack of a systematic quality assessment process for
hood. Incompleteness of data is clearly an issue and might be registries, which has yet to be developed, this study has followed the
explained by the fact that users indicated that patient registration recommendations of evaluation according to quality indicators and
was mostly done when they remembered to do so, and there is no dimensions, site monitoring, and a questionnaire [18]. Furthermore,
reliable notification to enter follow-up data other than a note in the we have conducted several methods previously described on how to
patient medical file, which, with a completion rate of around 60% conduct a registry quality assessment, including an intra- and inter-
per data element, is visibly unreliable and insufficient. Additionally, user variability study, providing feedback and recommendations,
the limited clinical relevance or predictive value of clinical outcome and writing the present data quality report [17]. Therefore, the
at one year follow-up presumably limits surgeons to complete the present quality assessment encompasses all the available methods
data. Therefore, expanding the follow-up period not only has more to evaluate the ARM-Net registry appropriately.
clinical relevance, but surgeons may also be more inclined to Recommendations for improving the quality of the ARM-Net
complete these data. registry and other rare disease registries.
The user questionnaire aimed to gain an insight on changes that Three areas of the ARM-Net registry were identified as requiring
users would like to see and how they experience patient registra- improvement: 1) structure of data collection, 2) completion of data,
tion, but also to clarify certain incongruencies in the collected data. and 3) clinical value of data. Firstly, addition and removal of data
Both user satisfaction and ease of use were highly regarded by the elements, expansion of answer options, nested further specification
users. Interestingly, it was apparent that many users preferred the items dependent on selected answer options, default answers or
availability of free text elements to continue in the future. However, error messages when items left blank to prevent missing data, and
from a research and data management perspective, free text ele- a reduction of free text fields should be considered. Data collected
ments are difficult to analyze and require extensive data cleaning. via free text elements should be evaluated to create additional
This demonstrates that there is a discrepancy between what the answer options and elements, as the existing elements and answer
ARM-Net registry users, mostly pediatric surgeons or otherwise options seem not to be satisfactorily sufficient. Data elements with
clinical staff, ideally prefer to collect, and what is ideal from a data large amounts of missing data or frequent ‘Unknown’ answers
research perspective. Furthermore, it was also clear that the users should be considered for removal. To decrease the data cleaning
would like to move towards a more clinically oriented and patient- burden and improve quality, free text fields should only be available
centered registry from their preference to remove dysmorphia- and when additional specifications are expected to be valuable.
hereditary-related items and start collecting PROMs. Engagement Expanding answers options and minimizing the availability of free
of patients will enhance the registry's scope and longevity, and text fields should also reduce intra- and inter-user variation and
provide valuable insight into a patient's life [19]. PROMs are para- leaves less room for interpretation differences. Furthermore, sys-
mount to consider when aiming to improve clinical care, in line tem-automated data accuracy checks, such as calculations of sur-
with the users' preferred future purpose of the ARM-Net registry. gery date after date of birth, should be implemented.
The current study has several limitations. Firstly, the evaluation Secondly, completeness of follow-up data entry should be
of the structure of the registry, its data elements, and the collected improved. One-year follow-up data entry might not have reached
data was performed based on the available literature, rather than 100% completion rate as there may have been patients that have
an existing methodology, which has yet to be developed. Consid- undergone their reconstructive surgery less than one year ago. Yet,
ering that patient registries can vary widely in their covered con- more than 70% of the patients in this analysis whose records have
dition, purpose, structure, and lifetime, no single method can missing data have undergone their reconstruction before 2020,
encompass all aspects of a registry evaluation. However, the critical suggesting that the time since reconstruction does not explain low
analysis in the current study is based on well-recognized compo- completion rate. Another, more plausible explanation, is that users
nents of data quality assessments in patient registries, such as must independently remember to enter one-year follow-up data,
quality dimensions of accuracy, validity, completeness, consistency, without proper notification. To improve completeness for one-year
usefulness, and prevention of duplicate entries [4,18e22]. A second follow-up, users should be automatically reminded by the EDC
limitation is that the intra- and inter-user variability study only system if data is required a year after reconstruction. Although data
included two different users, with different levels of training, for the remaining (closed-ended) elements in the registry was
registering a small sample of 10 patients. Ideally, quality nearly complete, overall completeness should be improved by
 I.C. Hageman, H.J.J. van der Steeg, E. Jenetzky et al. / Journal of Pediatric Surgery 58 (2023) 1921e1928 1927

making the appropriate data elements mandatory to be filled in Foundation and the Australian Government. Dr Isabel Hageman is
once starting the registration of a patient. supported by the Academy Ter Meulen grant of the Academy
Thirdly, taking feedback from the users into account, it is rec- Medical Sciences Fund of the Royal Netherlands Academy of Arts &
ommended to conduct a critical evaluation of the clinical value of Sciences (KNAW).
the current data elements. For example, current data on diagnostic
tests, and whether additional diagnostic procedures (e.g., voiding Declaration of competing interest
cystourethrogram, cystoscopy, echocardiogram) should be consid-
ered. Additionally, the registry should improve the structure of data None.
collection on whether the ARM is part of a syndrome. The list of
syndromes should be elaborated and these, as well as the indi-
Acknowledgements
vidual ARM types, should have corresponding standardized
ontology codes, such as Orphanet Rare Disease Ontology (ORDO),
Full members of the ARM-Net consortium not mentioned as
Online Mendelian Inheritance in Man (OMIM), or the International
coauthors by person: Dalia Aminoff, AIMAR, patient organization,
Classification of Diseases (ICD) [24e26]. Furthermore, to improve
Italy; Eva Amerstorfer and Holger Till, Medical University of Graz,
the value of the follow-up data and based on user suggestions, the
Austria; Piero Bagolan and Barbara Iacobelli, Ospedale Bambino
follow-up period should be extended to at least five years but
Gesù, Rome, Italy; Hakan Çavuşog
lu and Onur Ozen, Gazi Univer-
ideally to lifelong, as ARM is a condition that continues to affect
sity, Faculty of Medicine, Ankara, Turkey; Stefan Deluggi and
patients throughout their lives. Particularly because transitional,
Johanna Ludwiczek, Kepler University Hospital GmbH, Linz,
adult, and old-age outcome data are extremely scarce or incom- _
Austria; Emre Divarci, Ege University Medical School, Izmir, Turkey;
plete for rare congenital diseases such as ARM. Long-term out-
~ 
María Fanjul, Hospital Gregorio Maranon, Madrid, Spain; Francesco
comes and longitudinal data collection should be facilitated
Fascetti-Leon, University of Padua, Italy; Araceli García Va zquez,
through standardized case report forms at specific, predetermined
Hospital 12 de Octubre Madrid, Spain; Carlos Gine , Hospital Vall
time points, with automated reminders before and, if registration
d’Hebron, Barcelona, Spain; Ramon Gorter and Justin de Jong,
not completed, warning reminders after.
Amsterdam University Medical Centers, The Netherlands; Jan
It is not only the ARM-Net registry that can benefit from this
Goseman and Martin Lacher, University Hospital Leipzig, Germany;
quality assessment, as there are many suggestions that are appli-
Caterina Grano, Sapienza University of Rome, Italy; Sabine
cable to all rare disease patient registries. A recent systematic re-
Grasshoff-Derr, Buergerhospital and Clementine Childrens Hospi-
view [27] highlighted that many registries struggle with quality
tal, Frankfurt, Germany; Michel Haanen, VA, Patient organization,
management and maintenance. Like the suggestions resulting from
the Netherlands; Ernesto Leva and Anna Morandi, Fondazione
this study, protocolized periodical monitoring procedures, evalua-
IRCCS Ca' Granda Ospedale Maggiore Policlinico, Milano, Italy;
tions of user feedback, implementation of coding languages, and
Gabriele Lisi, Santo Spirito Civil Hospital, Pescara, Italy; Igor
mandatory fields to promote completeness are amongst the rec-
Makedonsky, Children's Hospital Dnepropetrovsk, Ukraine; Carlo
ommendations to improve existing registries. Additionally, mone-
Marcelis, Radboudumc, Nijmegen, The Netherlands; Paola Midrio,
tary incentives per registration, revision of research aims, and
Ca Foncello Hospital, Treviso, Italy; Marc Miserez, University Hos-
securing long-term sources of funding are important aspects to
pitals of the Katholieke Universiteit Leuven, Belgium; Mazeena
maintenance strategy [27].
Mohideen, SoMA Austria, patient organization, Austria; Alessio
Most importantly, registry developers and maintainers should
PiniPrato, Azienda Ospedaliera Nazionale SS A. e B. e Cesare Arrigo,
recognize that no registry will be perfect from its establishment,
Italy; Carlos Reck-Burneo, Medical University Wien, Austria; Heiko
and they should continuously be evaluated for improvement.
Reutter, University of Bonn, Germany; Stephan Rohleder, Medical
Registries, even if they have been running for more than a decade
University Hospital, Mainz, Germany; Inbal Samuk, Schneider
such as the ARM-Net registry, are malleable and should consider
Children's Medical Center, Petah Tikva, Israel; Eberhard Schmie-
changes and updates resulting from periodical quality assessments
deke, Klinikum Bremen Mitte, Bremen, Germany; Nicole Schwarzer,
to remain relevant. Sustainability of registries is key and with this
SoMA e.V., patient organization, Germany; Pim Sloots, Erasmus
study, the authors call for other rare disease patient registries to
MC-Sophia Children's Hospital Rotterdam, The Netherlands; Per-
take example to enhancing the small, but impactful field of rare
€ m, Lund University, Skane University Hospital, Lund,
nilla Stenstro
disease research.
Sweden; Chris Verhaak, Amalia Children's Hospital, Radboudumc
Nijmegen, The Netherlands; Alejandra Vilanova-Sa nchez, Univer-
5. Conclusion
sity Hospital La Paz, Madrid, Spain; Patrick Volk, University Hospital
Heidelberg, Heidelberg, Germany; Marieke Witvliet, Wilhelmina
The ARM-Net registry collects information that is undeniably
Children's Hospital, Utrecht, The Netherlands.
very valuable demonstrated by its consensus statements and
publications. However, as the registry has outgrown its original
purpose, data quality remains a challenge with vulnerability to Appendix A. Supplementary data
error and tendency to intra- and inter-user variability. Neverthe-
less, users were satisfied with the ARM-Net registry. This quality Supplementary data to this article can be found online at
assessment resulted in suggestions for improvement for the ARM- https://doi.org/10.1016/j.jpedsurg.2023.02.049.
Net registry as well as other rare disease patient registries in gen-
eral. Periodical critical (self-) evaluation is key to continuously References
improving data quality in the aspiration for a registry to be sus-
[1] Orphanet Report Series. Prevalence and incidence of rare diseases: biblio-
tainable and remain relevant for future research and clinical care.
graphic data. Rare Diseases collection; 2022. Available from: http://www.
orpha.net/orphacom/cahiers/docs/GB/Prevalence_of_rare_diseases_by_
Disclosure of funding diseases.pdf.
[2] Kancherla V, Sundar M, Tandaki L, Lux A, Bakker MK, Bergman JE, et al.
Prevalence and mortality among children with anorectal malformation: a
Professor King is generously supported in his role as an Aca- multi-country analysis. Birth Defects Research;n/a(n/a):1e15. doi:10.1002/
demic Paediatric Surgeon by the Royal Children's Hospital bdr2.2129.
1928 I.C. Hageman, H.J.J. van der Steeg, E. Jenetzky et al. / Journal of Pediatric Surgery 58 (2023) 1921e1928

[3] Berger A, Rustemeier AK, Gobel J, Kadioglu D, Britz V, Schubert K, et al. How to fistula: an ARM-Net consortium study. J Pediatr Surg 2022;57(9):89e96.
design a registry for undiagnosed patients in the framework of rare disease https://doi.org/10.1016/j.jpedsurg.2022.02.015.
diagnosis: suggestions on software, data set and coding system. Orphanet J [16] Gliklich RE, Dreyer NA, Leavy MB. Modifying and stopping registries. Regis-
Rare Dis 2021;16(1):198. https://doi.org/10.1186/s13023-021-01831-3. tries for evaluating patient outcomes: a user's guide. 3rd ed. Rockville, MD,
[4] Kodra Y, Weinbach J, Posada-de-la-Paz M, Coi A, Lemonnier SL, van United States: Agency for Healthcare Research and Quality; 2014.
Enckevort D, et al. Recommendations for improving the quality of rare disease [17] Arts DG, De Keizer NF, Scheffer GJ. Defining and improving data quality in
registries. Int J Environ Res Public Health 2018;15(8). https://doi.org/10.3390/ medical registries: a literature review, case study, and generic framework.
ijerph15081644. J Am Med Inform Assoc 2002;9(6):600e11. https://doi.org/10.1197/
[5] de Blaauw I, Wijers CH, Schmiedeke E, Holland-Cunz S, Gamba P, Marcelis CL, jamia.m1087.
et al. First results of a European multi-center registry of patients with ano- [18] Kodra Y, Posada de la Paz M, Coi A, Santoro M, Bianchi F, Ahmed F, et al. Data
rectal malformations. J Pediatr Surg 2013;48(12):2530e5. https://doi.org/ quality in rare diseases registries. In: Posada de la Paz M, Taruscio D, Groft SC,
10.1016/j.jpedsurg.2013.07.022. editors. Rare diseases epidemiology: update and overview. Cham: Springer
[6] Amerstorfer EE, Grano C, Verhaak C, García-Vasquez A, Miserez M, Radleff- International Publishing; 2017. p. 149e64. https://doi.org/10.1007/978-3-
Schlimme A, et al. What do pediatric surgeons think about sexual issues in 319-67144-4_8.
dealing with patients with anorectal malformations? The ARM-Net con- [19] Gliklich RE, Leavy MB, Dreyer NA. Registries for evaluating patient outcomes:
sortium members' opinion. Pediatr Surg Int 2019;35(9):935e43. https:// a user's guide. 4th ed. Rockville, MD, United States: Agency for Healthcare
doi.org/10.1007/s00383-019-04506-0. Research and Quality; 2020. Report No.: 19(20)-EHC020.
[7] Amerstorfer EE, Schmiedeke E, Samuk I, Sloots CEJ, van Rooij I, Jenetzky E, [20] The six primary dimensions for data quality assessment: defining data quality
et al. Clinical differentiation between a normal anus, anterior anus, congenital dimensions. The DAMA UK working group. vailable from: https://www.
anal stenosis, and perineal fistula: definitions and consequences-the ARM-net dqglobal.com/wp-content/uploads/2020/11/THE-SIX-PRIMARY-
consortium consensus. Children 2022;9(6). https://doi.org/10.3390/ DIMENSIONS-FOR-DATA-QUALITY-ASSESSMENT.pdf.
children9060831. [21] Zalatel M, Kralj M. Methodological guidelines and recommendations for
[8] van der Steeg HJ, Schmiedeke E, Bagolan P, Broens P, Demirogullari B, Garcia- efficient and rational governance of patient registries. Ljubljana, Slovenia:
Vazquez A, et al. European consensus meeting of ARM-Net members con- National Institute of Public Health; 2015. p. 83e106. Available from: https://
cerning diagnosis and early management of newborns with anorectal mal- health.ec.europa.eu/system/files/2016-11/patient_registries_guidelines_en_0.
formations. Tech Coloproctol 2015;19(3):181e5. https://doi.org/10.1007/ pdf.
s10151-015-1267-8. [22] Discussion paper: use of patient disease registries for regulatory purposes e
[9] Jenetzky E, van Rooij IA, Aminoff D, Schwarzer N, Reutter H, Schmiedeke E, et al. methodological and operational considerations. London, United Kingdom:
The challenges of the European anorectal malformations-net registry. Eur J The Cross-Committee Task Force on Patient Registries, European Medicines
Pediatr Surg 2015;25(6):481e7. https://doi.org/10.1055/s-0035-1569149. Agency: 23-4. Available from: https://www.ema.europa.eu/en/documents/
[10] Midrio P, van Rooij I, Brisighelli G, Garcia A, Fanjul M, Broens P, et al. Inter- other/discussion-paper-use-patient-disease-registries-regulatory-purposes-
and intraobserver variation in the assessment of preoperative colostograms in methodological-operational_en.docx.
male anorectal malformations: an ARM-net consortium survey. Front Pediatr [23] Holschneider A, Hutson J, Pen ~ a A, Beket E, Chatterjee S, Coran A, et al. Pre-
2020;8:571. https://doi.org/10.3389/fped.2020.00571. liminary report on the international conference for the development of
[11] Morandi A, Fanjul M, Iacobelli BD, Samuk I, Aminoff D, Midrio P, et al. Uro- standards for the treatment of anorectal malformations. J Pediatr Surg
logical impact of epididymo-orchitis in patients with anorectal malformation: 2005;40(10):1521e6. https://doi.org/10.1016/j.jpedsurg.2005.08.002.
an ARM-net consortium study. Eur J Pediatr Surg 2022. https://doi.org/ [24] Vasant D, Chanas L, Malone J, Hanauer M, Olry A, Jupp S, et al. ORDO: an
10.1055/s-0042-1742300. ontology connecting rare disease, epidemiology and genetic data. 2014.
[12] Samuk I, Amerstorfer EE, Fanjul M, Iacobelli BD, Lisi G, Midrio P, et al. Perineal Available from: https://www.researchgate.net/publication/281824026_
groove: an anorectal malformation network, consortium study. J Pediatr ORDO_An_Ontology_Connecting_Rare_Disease_Epidemiology_and_Genetic_
2020;222:207e12. https://doi.org/10.1016/j.jpeds.2020.03.026. Data.
[13] Samuk I, Gine C, de Blaauw I, Morandi A, Stenstrom P, Giuliani S, et al. Ano- [25] Hamosh A, Scott AF, Amberger JS, Bocchini CA, McKusick VA. Online Men-
rectal malformations and perineal hemangiomas: the Arm-Net Consortium delian Inheritance in Man (OMIM), a knowledgebase of human genes and
experience. J Pediatr Surg 2019;54(10):1993e7. https://doi.org/10.1016/ genetic disorders. Nucleic Acids Res 2005;33(Database issue):D514e7.
j.jpedsurg.2018.10.111. https://doi.org/10.1093/nar/gki033.
[14] van der Steeg HJJ, van Rooij I, Iacobelli BD, Sloots CEJ, Leva E, Broens P, et al. [26] World Health Organization. International classification of diseases, eleventh
The impact of perioperative care on complications and short term outcome in revision (ICD-11). Geneva: World Health Organization; 2019/2021. Available
ARM type rectovestibular fistula: an ARM-Net consortium study. J Pediatr from: https://icd.who.int/browse11.
Surg 2019;54(8):1595e600. https://doi.org/10.1016/j.jpedsurg.2019.03.008. [27] Hageman IC, van Rooij IALM, de Blaauw I, Trajanovska M, King SK. A sys-
[15] van der Steeg HJJ, van Rooij I, Iacobelli BD, Sloots CEJ, Morandi A, Broens PMA, tematic overview of rare disease patient registries: challenges in design,
et al. Bowel function and associated risk factors at preschool and early quality management, and maintenance. Orphanet J Rare Dis. 2023 May
childhood age in children with anorectal malformation type rectovestibular 5;18(1):106. https://doi.org/10.1186/s13023-023-02719-0.