Universal Access in the Information Society

https://doi.org/10.1007/s10209-025-01194-7

LONG PAPER

Combined accessibility validation and monitoring of web sites

and PDF documents
Nicola Iannuzzi1 · Marco Manca1 · Fabio Paternò1 · Carmen Santoro1

Accepted: 27 January 2025

© The Author(s) 2025

Abstract
Accessibility validation of online digital content through automatic tools has been addressed in limited terms. There is a need
for a holistic approach to accessibility validation, able to align well with the goals of public organisations and accessibility
authorities. In this perspective, one main issue is that usually such tools aim to assess either the web content, or the PDF
documents, but not both. However, users need that all the content be accessible, regardless of the format used, thus it would
be helpful to have tools able to perform a combined analysis of both web pages, and PDF files connected with such pages.
This combined approach to accessibility validation, beyond providing a more complete and coherent view on the accessibil-
ity supported for everyone, would also be important for accessibility authorities (who must monitor the state of accessibility
on a large scale, e.g., to comply with the EU WAD Directive), as well as for public organisations (since they could incur in
legal risks and potential lawsuits, if only web pages are compliant). Indeed, a combined accessibility analysis would help
them to identify the sites that need more interventions and would also be useful more generally to stimulate web developers
and content providers to pay attention to both aspects. In this paper, we present how a tool that originally supported Web
accessibility validation only, has been extended to include in the supported accessibility analysis also PDF files, to provide
a more comprehensive assessment of the evaluated content. We describe the tasks it supports, the possible use cases, as well
as some results obtained in a large-scale combined validation (Web and PDF content), carried out on Italian public service
websites. The paper also reports the results of a user study carried out to understand the usability of the features that were
added to the tool to support the validation of the accessibility of both web and PDF content.

Keywords Accessibility · Automatic validation tools · Accessibility monitoring

1 Introduction services, with the W3C WCAG guidelines indicating how

accessibility should be supported, and unfortunately many
The importance of digital accessibility is becoming widely violations of such guidelines can still be found. Indeed, a
recognised by national and international legislation due recent report by WebAIM1 has analysed the accessibility of
to its high impact on users, who should be able to access the home pages for the top popular 1,000,000 web sites and
digital information regardless of their abilities and skills. found that users with disabilities would expect to encounter
The Web is the most common channel for accessing public errors on 1 in every 21 home page elements.
In general, a thorough accessibility evaluation can be
very complex to perform, given the large number of ele-
* Nicola Iannuzzi ments that make up websites, and the continuously increas-
[email protected] ing number of success criteria and techniques to check. To
Marco Manca address such complexity, interest in automatic support for
[email protected] accessibility validation is increasing. Automatic tools can
Fabio Paternò help obtain more complete and consistent evaluations, which
[email protected] are currently in high demand also because developers are
Carmen Santoro often unaware of possible incorrect or inconsistent behav-
[email protected] iour of their applications from an accessibility perspective
1 1
CNR-ISTI, HIIS Laboratory, Pisa, Italy https://​webaim.​org/​proje​cts/​milli​on/

Vol.:(0123456789)

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 Universal Access in the Information Society

[15]. However, it is worth noting that automatic tools cannot these two types of content would be very useful for checking
validate all the possible accessibility aspects, although they and monitoring the accessibility levels of public services in a
can provide indications for cases when a manual interven- more comprehensive manner, offering an integrated analysis
tion is necessary. The use of automatic tools can also better encompassing both aspects in one place. A recent analysis
support the activity of monitoring the accessibility levels [8] on the accessibility of some international healthcare Web
of public services which, as highlighted by the WAD (Web services (carried out by using manual checks) identified five
Accessibility Directive) EU directive [7], is also fundamen- main problems in the web pages and in the resources linked
tal to identify the areas that require more attention. within them: inaccessible images, challenges accessing addi-
In this perspective, one aspect that has not yet received tional resources, poor structural formatting, lack of tagging
sufficient attention from Web validation tools is the pos- in PDFs, and minimal colour contrast. That study found
sibility of supporting a combined validation of both Web that only 4.5% of the selected PDF documents were tagged.
pages and associated PDF (Portable Document Format) Thus, most documents were not compatible with text-to-
documents. This is important because, often, for disparate speech software and thus not accessible for several catego-
purposes (e.g., announcements, certificates, documents ries of users. A previous study [1] considered web sites of
issued, forms patients to fill in, newsletters, annual reports, universities in Latin America and checked their PDF docu-
and order forms), public administrations provide their ser- ments according to the WCAG 2.0 guidelines (and using the
vices on the web in association with related PDF documents. PDF Accessibility Checker 2.0) and obtained similar results
Thus, especially for the web sites of these organisations, not indicating that universities were not sufficiently committed
only web pages but also PDF documents must be accessible in providing accessible documents.
to people, regardless of their abilities, otherwise they will be Regarding the possibility of supporting the validation
excluded from accessing important information. of both Web and PDF resources, the Tingtun Checker pro-
There are already tools that allow accessibility valida- vided some early support addressing both types of contents.
tion of PDF documents (e.g., Adobe Acrobat Pro), but they However, such support is rather separated within the tool
do not address online Web content, whereas, conversely, since users must indicate either the Web page or the PDF
most Web accessibility validators currently consider only document to validate. Another limited support in this direc-
web pages in their analysis. Regarding PDF accessibility, tion is given by the WordPress Accessibility Checker plugin
the most widely recognized standards are the one given by (https://w
​ ordpr​ ess.o​ rg/p​ lugin​ s/a​ ccess​ ibili​ ty-c​ hecke​ r/), which
the Web Content Accessibility Guidelines (WCAG), and can find the links to PDF documents in a web document and
the PDF/UA (Universal Accessibility) standard provided by generate associated warnings indicating the need for acces-
ISO (ISO 14289). The W3C WCAG guidelines have a broad sibility validations and display the related success criteria.
scope and consider both web and non-web content (includ- However, with such tools, it is not possible to perform in a
ing PDF), while PDF/UA focuses specifically on the acces- single request the validation of a given web page and the
sibility of PDF documents, by providing technical specifi- PDF documents linked to it, which is especially important
cations for creating accessible PDFs, covering areas such when the goal is to monitor the level of accessibility of pub-
as document structure, metadata, colour, fonts, multimedia, lic services.
and forms. Unfortunately, the accessibility of the PDF docu- The main contribution of this paper is then to put for-
ments available on the Internet has often been neglected. A ward a solution to such issues, by presenting the design and
study [14] found that in 200 articles in PDF format published implementation of a tool supporting the integrated validation
by four journals from 2009 to 2013, 97% of them did not of Web pages and PDF content, also showing its applica-
provide alternative texts for images, and only 4.5% of the tion and utility in a real-world scenario. Our main research
articles were produced as tagged PDF documents. Although questions are whether this tool is usable by novice users
the most recent legislation in several countries requires that with minimal instructions, and whether this tool can also
educational and research documents be accessible, very be used in real-world scenarios such as the monitoring of
often these legal requirements are not enforced. Thus, there Web + PDF resources in a large-scale validation carried out
is the need to support the identification of problematic cases. at a national level.
In this regard, we think that a unified approach address- In the paper we describe how we have designed the sup-
ing at the same time both types of content (Web and PDF) port for such combined analysis in the MAUVE++ tool,
can have several advantages. For instance, not only it could which originally focused only on the validation of web
better support consistency (and clarity) in both the interpre- pages, and now it has been extended to include in the analy-
tation of the evaluation criteria and in the provision of the sis also PDF resources. After discussing some related work
reported outcomes, but it can also simplify the workflow of in the next section, we then present the main approach
its users (since they need to use just a single tool and not exploited in the tool, its implementation, and the feedback
multiple ones). More importantly, a combined analysis of that we gathered in a user test focusing on its usability. We

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Universal Access in the Information Society

also describe the results of a large-scale analysis carried out and mobile applications for this purpose. Indeed, the WAD
on Italian public organisations’ websites, using the presented requires that Member states must evaluate their websites
tool, and considering both Web pages and PDF files. every three months and produce accessibility monitoring
reports; regarding the methodology to adopt, they must do
both automatic and manual accessibility evaluations. This
2 Related work stimulated further interest in automatic support for acces-
sibility validation, given the large number of checks to per-
The increasing amount of digital content and the need to form. Thus, some large-scale accessibility evaluation studies
check its accessibility (also enforced by laws) has pushed have started to be carried out, in a few cases involving even
the evolution of accessibility validators, which also need to mobile apps [18].
consider the ability to monitor over time the level of acces- In this regard, Martins and Duarte have put forward
sibility of the considered applications. Some tools [5] have two contributions. One [12] considered web accessibility
been proposed as post-processing software tools to make metrics in large scale studies. Such metrics are quantita-
PDF documents accessible, but they suffer from some usa- tive indicators obtained through specific formulas, which
bility issues [17]. Even though some of such issues have are applied using data provided by accessibility evaluations,
been recently addressed in Ally [16], a tool to assist content and are aimed at synthesising the accessibility level of a
creators in remediating their PDF files to improve acces- web resource through a specific value. In particular, for their
sibility, such tools still do not support combined views of analysis, they considered and compared eleven web accessi-
accessibility of web and PDF content. bility metrics, computed using the QualWeb tool, and a sam-
By filtering the initial list of 129 tools for accessibility ple of around three million web pages, taken from an open
validation provided by the W3C [21] (retrieved on January corpus of web data. The other one [13] focused on the analy-
2024) to select only those offering analysis for both web sis of the technologies used in Web sites implementation.
pages and PDF documents, only 20 tools were identified, However, none of these two studies considered the linked
which unfortunately did not seem well suited to meeting the PDF documents in the accessibility validation. Still regard-
goals set by this work for several reasons: ing large scale studies, Carvalho et al. [4] investigated the
impact of optimising the page selection processes in large-
• Lack of PDF analysis information. Some tools do not scale web accessibility evaluations, but they did not consider
explicitly mention PDF document analysis on their PDF resources. Also, a recent analysis of the impact of the
respective web pages, such as BOIA, Accessin, Odellus; European WAD directive on how to perform the monitoring
• Unavailable tools. Some tools are no longer available, of accessibility at a national level [6] did not find solutions
such as Crownpeak, User1st; for combined Web and PDF analysis.
• Only specific assistive technologies support is provided.
Some tools offer support for access only through assistive
technologies (such as screen readers or screen magni- 3 Approach
fiers) to check accessibility, but do not provide direct
analysis for PDF documents and web pages, such as Our work has been stimulated by our direct experience in
Assistivlabs, TPGi ARC Platform—JAWS Inspect; validating Web content, discussions with disabled users, and
• Paid services. Several tools are not publicly available and with stakeholders belonging to the Italian national author-
thereby require paid access, such as Siteimprove, Acces- ity for accessibility monitoring (Agency for Digital Italy,
sibilityOz; AgID). In general, it emerged that when disabled users
• Lack of combined validation support. Even though some access Web content, especially when this involves public
tools provide validation support for both web pages and websites and services, they need all parts to be accessible
PDF resources (such as AChecks), unfortunately they (both the Web pages and the documents linked to them).
do not offer combined validation services, which means Such a need has stimulated the national authority to extend
that the two validations (Web and PDF) are performed the accessibility monitoring activities to PDF documents,
in a separate manner, without showing the relationships which are the most used non-web-linked documents.
between the two types of contents, which is necessary for We organised a webinar on the topic of accessibility veri-
supporting an integrated analysis. fication of web sites, targeting public administrations. On
this occasion, we administered a questionnaire to a group
The importance of making digital content accessible of people working in public administration and interested
for all has been recognised by legislation of several coun- in accessibility.
tries, which also often indicate (e.g., the WAD Directive Fourteen persons (five females) answered the question-
[7]) the necessity to monitor the accessibility levels of web naire (average age: 50.8; st.dev = 7,5). They self-identified

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 Universal Access in the Information Society

as Accessibility Experts (six persons) or Web Developers the accessibility of Italian public administration websites:
(three persons); most of them (nine persons) had used at at some point, they asked to extend the analysis also to the
least one tool for accessibility validation (e.g. MAUVE++, PDF documents linked to web pages, since they noticed that
Site Improve, PAVE, W3C validator). We also asked them several times they were found to be lacking in accessibility.
about the frequency with which they used such tools: among Since PDF is one of the most widely used formats for dis-
the nine users who replied to this question, five participants seminating information on the web, and taking into account
used it at least once per month, two users once per week. We the users’ requests and requirements, the new version of the
asked their feedback on several aspects relevant to accessi- MAUVE++ validator aims to offer a more comprehensive
bility, using a scale from 1 to 5, where 1 means “not impor- and thorough analysis of the accessibility of resources found
tant at all” or “not useful at all” and 5 “very important” or on websites, with specific attention to linked PDF docu-
“very useful”. Most of the participants (twelve users) believe ments. Finding and evaluating the PDF documents linked
that it is very important for Public Administrations to pro- to web pages decreases the manual efforts in searching for
vide accessible PDF documents linked through their web- related resources, and this allows users to focus directly on
sites. However, most participants (ten users) do not know the evaluation and enhancement of accessibility, ensuring
a publicly available tool that enables combined analysis quicker access to relevant information.
of web pages and associated PDF documents. Moreover, To achieve this, we identified several functionalities
only three participants used a tool for validating PDF files. (requirements) that the tool should support:
Of the total respondents, four users indicated that some
PDFs linked in the institutional sites are created by scan- • (R1) Evaluation of Single PDFs: Enabling a detailed and
ning paper-based documents: this method, when it does not specific analysis of single PDF documents, by evaluating
include the application of OCR technology (which converts their accessibility in accordance with the main relevant
visual materials into textual ones) would create inaccessible WCAG techniques;
documents (the scanned PDFs are just images, therefore they • (R2) Evaluation of Single Web Pages with Associated
are inaccessible to screen readers). PDFs: Providing an assessment of the accessibility of a
We also asked participants to assess four features that a single web page, also considering the PDF documents
comprehensive tool for evaluation of web pages and PDF linked to it. The goal is to verify both the accessibility
should support (using the same scale detailed before): of the web page and the associated PDF documents in a
single shot, also providing combined results;
• Single PDF evaluation (12 users judged it very useful); • (R3) Website-wide Accessibility Analysis with PDFs:
• Single web Page evaluation along with all PDF docu- Evaluating the accessibility of a larger number of pages,
ments linked within it (8 users judged it very useful, 5 up to an entire website, including the analysis of associ-
users useful); ated PDF documents found across the considered pages.
• Evaluate the whole website along with all PDF docu- This function should also offer an overview of accessibil-
ments linked in the discovered pages (for 10 users it is ity on a broader scale;
very useful, 3 of them consider it useful); • (R4) Continuous Monitoring: Supporting continuous
• Monitoring the accessibility of web pages and PDF docu- monitoring over time of validation results for both Web
ments over time (for 9 users it is very useful, for 4 users pages and PDF documents. This allows observing the
it is useful). evolution of PDF document and Web page accessibility
over time, providing a more comprehensive and continu-
Among future developments, participants indicated as ous view of performance trends;
“very important” two additional features: provide practical • (R5) Semi-Automatic Error Correction: Implement-
indication about how to solve accessibility issues within a ing a semi-automatic error correction system for errors
PDF document (13 of them consider it very important, 1 detected in PDF documents. The goal is to provide users
user declared it important); automatically fix accessibility with the ability to make corrections directly within the
issues of PDF documents linked to a Web Page (9 users: document while viewing the obtained results.
very important, 2 participants: important).
Thus, the answers provided by the responding partici-
pants confirm that the integration of web pages validation 4 User interface
with the analysis of PDF documents is a crucial and neces-
sary requirement in online content accessibility. Apart from MAUVE++ was initially developed to provide support for
the suggestions provided by those who filled in the question- Web accessibility validation. It has been extended to sup-
naire, we also received some requests in the same direction port the requirements identified in the previous section. For
from AgID, which was already using MAUVE++ to monitor this purpose, its user interface has been enhanced to provide

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Universal Access in the Information Society

three different ways to verify the accessibility of PDF docu- PDF documents linked to the web page. Thus, in this case,
ments, also providing support for analysis and resolution of the validation results regard both the web page and the
detected errors. PDF documents, and they are shown in the ‘Web Page
Results’ and ‘PDF Docs Results’ tabs. The combined
4.1 Evaluation of single PDFs accessibility percentage obtained from the results of the
web page and linked PDF documents is shown at the page
This mode enables users to analyse a single PDF document top (see Fig. 2).
(requirement 1). To specify the PDF to analyse, the tool (see
Fig. 1) allows users either to indicate the PDF's URL, or to
directly upload the PDF file stored in the user’s file system 4.3 Website‑wide accessibility analysis with PDFs
or available in a web site private area. Once the analysis is
complete, the results are displayed in detail, supporting the The tool provides the possibility to create a “Project” for
correction of some errors. the validation of multiple web pages and the PDF docu-
From a privacy and copyright perspective, the PDF files ments linked to those web pages (requirement 3). During
are retained in memory only for the duration of the analysis, the project creation, users specify the seed URL (usually
and once the analysis is complete, the documents are not the home page), which is sent to the crawler. During the
stored or further processed by the tool. crawling process, the tool starts from the seed URL and
analyses all the links found on the page. A web page link
4.2 Evaluation of single web pages with associated is added to the list of the discovered pages only if its URL
PDFs belongs to the same domain as the seed, while all the links
to PDF documents found are added to the list of discovered
Within the context of a single web page accessibility vali- PDF documents regardless of their domain. Figure 3 shows
dation, the tool provides the option to discover and vali- a project example, on the right the list of URLs, where
date the PDF documents linked to the page (requirement for each page the number of PDF documents linked to it
2). After indicating the Web page to analyse, users can is displayed. In this way, we make the user immediately
exploit this feature by selecting a specific checkbox, which aware of the PDF content associated with the considered
activates the validator to search for and then examine the Web pages.

Fig. 1  User interface for evaluating a single PDF

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 Universal Access in the Information Society

Fig. 2  Visualisation of Single Web Pages with PDFs Results

Fig. 3  Project info visualisation

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Universal Access in the Information Society

4.4 Continuous monitoring 4.5 Semi‑automatic error correction

MAUVE++ supports the possibility of multiple audits at One further possibility provided by the tool is to present the
different times over the same set of discovered web pages results of the evaluation analysis of a PDF document in a
[9] and PDF documents (Fig. 3). In this way, users can clear manner, offering users the opportunity to understand
monitor the evolution of accessibility over time (require- the errors, read the guideline documentation, fix the issues
ment 4) for a set of web pages, also considering their (Requirement 5), and upload the corrected PDF intuitively
associated PDF resources. The list of the evaluated web and quickly. The visualisation of the evaluation results of the
pages shows an icon beside each webpage URL, indicat- PDF documents is the same for all types of analyses (single
ing the presence of PDF documents linked to it. On the pdf, single page, projects) (Fig. 5). It is designed to offer a
page providing details about every single audit, specific consistent, clear and detailed overview of detected errors,
reports are provided for each web page considered for achieved successes, and available corrective actions. This
accessibility audit and for each analysed PDF document visualisation focuses on specific details associated with indi-
linked to the considered page, detailing the number of vidual PDF documents and reports the results of the evalu-
techniques resulting in errors and successes. ation techniques.
The tool also provides information on the results of Figure 5 shows the tool page with the results obtained
the accessibility metrics over time (Fig. 4): the x-axis after validating a PDF file. In particular, in the top part of
indicates the various audits conducted (their dates) on all this page the file path of the considered PDF is indicated,
the resources in the project (Web pages and PDF docu- while the lower part of the screen is divided into two main
ments), while the y-axis shows the percentage value. The sections. On the left side, there is the PDF Display Area,
three bars for each audit represent (from left to right): the which is a component that loads and displays the content of
accessibility percentage of web pages, the completeness the analysed PDF. This component allows users to directly
of analysis for web resources, and the PDF accessibility view the PDF document without having to download or open
percentage. it separately. On the right side there is the Results Table.
In this table, the left column contains details regarding
the applied technique(s), including the WCAG guidelines,

Fig. 4  Trend according to MAUVE++ accessibility metrics (based on techniques)

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 Universal Access in the Information Society

Fig. 5  Visualisation of the evaluation results of a PDF document

success criteria, the name of the technique, a brief descrip- By clicking that button, a modal dialogue opens and guides
tion of the technique, and a link to the W3C documentation. the user through the error fixing process. For instance, for
The second column indicates the result obtained by applying the technique PDF18, which regards the document title
each evaluation technique to the PDF elements, which can specification, in case of error (no title is provided in the
be either ‘Pass’ or ‘Fail’. PDF document) the user can click on the ‘Quick fix’ button,
For some techniques, in case of errors, the tool shows a next specify the title and then save the updated document (in
‘Quick fix’ button, enabling the possibility of corrections. a local folder). After all the desired amendments have been

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Universal Access in the Information Society

applied, a button is displayed allowing the user to save (and In the new version of the tool (Fig. 7) we have introduced
download) the updated PDF document containing all the in the validation engine a new module (named PDFChecker)
fixes (see at the bottom of Fig. 6). to integrate the PDF evaluation support: this new module
It is worth noting that (as it happens for the web pages) implements some techniques related to PDFs, which are
also the PDF files are temporarily loaded in the tool’s mem- specified in WCAG guidelines. The PDF validation engine
ory only for the duration of the analysis. exploits the Apache PDFBox [2] library to access and ana-
lyse the PDF document structure and to verify the compli-
ance with the supported techniques. The Selenium service
4.6 Implementation [20] is used to retrieve the HTML code to be parsed. Follow-
ing the URLs provided by the crawler, the Web resources are
MAUVE++ supports the Web Content Accessibility Guide- loaded into a headless browser instance so that Web pages
lines (currently the WCAG v2.1) validation. The core archi- that exploit the client to render their content are also cor-
tecture of the tool [19] is composed of modules that exploit rectly loaded.
an XML specification of the techniques to validate. The tool The WCAG 2.1 specification includes 23 PDF techniques.
evolved over the years [3] by introducing different views MAUVE++ supports a subset of them (they are listed in
both for End Users (e.g. accessibility experts, showing to the following), selected as those considered representative
them the results of the validation in a graphical view through of frequent sources of accessibility problems, taking into
charts and tables) and for Web Developers (by reporting account the indications provided by relevant stakeholders for
results referring to the corresponding HTML code). the National Monitoring Accessibility process. Others will

Fig. 6  Example of Semi-

Automatic Error Correction for
PDF18 technique

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 Universal Access in the Information Society

Fig. 7  MAUVE++ application

architecture

be supported in the near future. Some controls that we found documents; such documents are saved as an image within the
useful in assessing certain aspects related to the accessibility PDF document and thus they are not accessible at all using
of PDF documents were indicated in the WCAG documen- assistive technologies, since they do not contain text that can
tation at the level of success criteria but not detailed at the be read by screen readers. This technique is particularly use-
more specific level of techniques, therefore, we refer to such ful to check whether a scanned PDF document is accessible.
techniques as “MAUVE++ Best Practices” and indicate the The implementation of MPDF2 technique exploits the PDF-
relevant WCAG success criterion. Such techniques are iden- Box library by trying to extract text from all pages belonging
tified starting with “MPDF” <x> , while the ones starting to the document. If at least one page contains some text, the
with “PDF” <x> are those corresponding to specific WCAG MAUVE++ considers the document accessible, and marks
2.1 techniques. the technique as passed. If no text is extracted from any
MPDF1 Use of tags to specify the logical structure of page, the technique fails.
the PDF (Referring to WCAG 2.1 Success Criterion 1.3.1). MPDF3 Allow the user to extract text and images from
It verifies the use of tags within PDF documents to define a the PDF document for accessibility purposes (Referring
logical structure. The presence of tags enhances accessibility to WCAG 2.1 Success Criterion 4.1.1). The PDF format
for users with visual impairments who utilise assistive tools. specification allows defining access permissions to a docu-
The algorithm analyses the PDF document's catalogue, a ment; among them, PDF document creators can set a prop-
dictionary that is the root node of the document, to verify erty expressing whether it is possible to extract text and
whether tag information is present. If the catalogue contains images for accessibility purposes. The implementation of
tag information, the technique is considered successful, and the MPDF3 technique verifies the restrictions within the
it will be marked as passed in the result list. Otherwise, if PDF document, specifically focusing on the entry related
tags are absent, the technique is set as failed. to extracting text and graphics for accessibility to visually
MPDF2 Searchable text in PDF document (Referring to impaired people. If this entry is set to “true”, the technique
WCAG 2.1 Success Criterion 1.1.1). It verifies the presence is considered passed, indicating success in the accessibility
of searchable text within the PDF document. Several PDFs check; if the property value is false, the technique is con-
available on the Internet are obtained by scanning paper sidered failed.

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Universal Access in the Information Society

PDF3 Ensuring correct tab and reading order in PDF the page content. The technique implementation verifies the
documents. The goal of this technique is to allow users to presence of a specified title in the document information
navigate through the content in an order logically consist- dictionary section of the PDF file. According to the PDF
ent with the meaning of the content. The validation verifies specification, a title can be set but it may not be displayed
the specification of the correct logical tabbing order within depending on the value of “/DisplayDocTitle” property set
the PDF document by checking each page of the PDF docu- within the document catalogue. If the title is specified and
ment for the presence of the ‘Tabs’ entry in the dictionary. the “/DisplayDocTitle” entry is present and active, the tech-
If the ‘Tabs’ entry is present and the associated value is ‘S’ nique is considered successful. However, if the title is not
(indicating that for documents with tags, the focus moves present or the “/DisplayDocTitle” property is deactivated,
according to the order of the tags), the technique consid- the technique registers a failure.
ers the correct tabulation order and marks it as successful.
Otherwise, if the ‘Tabs’ entry is not present or the associated
value is not ‘S’, the technique fails. The reading order must 5 Application in national monitoring
always be evaluated manually. of accessibility
PDF9 Checking headings to mark content in PDF docu-
ments. Sections of PDF documents should be marked with Under the aegis of a project dedicated to monitoring of
headings tags so that they can be recognized by assistive Accessibility of all Italian public administration websites,
technologies. Headings markup can be useful to indicate the we performed an integrated large-scale validation of both
beginning of a section, or sections containing important con- web pages and linked PDF documents, between November
tent, or navigational sections, etc. Assistive technologies can and December 2023. It has also been useful to verify the
navigate and directly move towards the appropriate heading, scalability of the solution proposed.
thus tagged elements help to improve the accessibility. The The source for the national monitoring is a public data-
PDF9 technique implementation verifies the presence of at base called IPA (Index of Digital Domains of Public Admin-
least one correctly marked header tag within the document. istration) along with several websites of Large Companies
While executing this technique, the structural elements of extracted from the Accessibility Declaration Database. The
the PDF document are analysed to search for “Sect” tags that spool provided to our tool is composed of 29.854 websites
identify sections and predefined header tags such as “H”, and for each one the home page URL that will be used as
“H1,” “H2,” “H3,” “H4,” “H5,” and “H6” within them. If seed for the crawler is provided. For additional information
all the identified sections have headers marked with appro- about the whole validation process see [10].
priate tags or, if there are no sections (“/Sect”) but at least The crawler ended its process for 26.468 websites
one header tag is present within the document, the verifica- (88.66%), while for 3.386 websites (11.34%) an exception
tion of the technique results in a success. If the document was thrown during the crawling process. The most common
does not contain headers, the technique records a failure by exceptions involved (Fig. 8):
adding the corresponding information to the list of negative
verifications (failures). • Websites that no longer exist (1.842 Name Not Resolved
PDF16 Setting the default language using the /Lang Exceptions, 54.4% of exceptions, 6.17% of the total web-
entry in the document catalogue of a PDF document. The sites);
goal of this technique is to highlight the importance of the • Existing websites but the home page URL provided is not
Lang attribute within PDFs documents; this attribute helps found (215 Page not found exceptions—error 404, 6.35%
assistive technologies to more properly render a text or load of exceptions, 0.72% of the total websites);
the correct pronunciation rules for screen readers and vocal • Pages not accessible (81 unauthorised exception—error
assistance. The implementation verifies the presence of the 403, 2.39% of exceptions, 0.27% of the total websites);
default language definition within the PDF document's cata- • 230 websites that failed to load (internal server excep-
logue. If the verification of this entry in the PDF document's tion—error 5XX, 6.79% of exceptions, 0.77% of the total
catalogue is successful, the technique is considered success- websites);
ful, if the "/Lang" entry is not present or not defined, the • 185 Website that failed to load (timeout exception, 5,46%
technique records a failure and adds the corresponding info of exceptions, 0,62% of the total websites).
to the list of negative checks (failures).
PDF18 Specifying the document title using the Title Thus, the crawling process provided a list of 24.958
entry in the document information dictionary of a PDF websites having at least 10 pages found. Regarding the
document: the goal of this technique is to show a descrip- PDF documents, the crawler limited the list to max 50 doc-
tive title for a PDF document; the title should identify the uments per site. MAUVE++ correctly evaluated 24.576
current document, so that users have not to read or interpret sites (98.46%) of the 24.958 provided by the crawler; and

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 Universal Access in the Information Society

Fig. 8  Crawling exceptions

types

among them, for 24.377 sites it evaluated at least 10 pages. assessed further emphasizes the critical role that non-web
The total number of the pages evaluated is 4.235.456. content plays in defining a website’s accessibility profile.
MAUVE++ marked 125 websites (0.5%) as “Blocked” Analysing these two components together—web pages
since their evaluation did not finish in 15 min; while for and PDF documents—offers a more holistic view of acces-
257 (1.03%) websites the evaluation process failed because sibility. Websites might comply with accessibility guidelines
an exception occurred (Fig. 9). for their HTML content, but non-compliant PDF documents
After MAUVE++ finishes a web site evaluation, it starts could still create significant accessibility issues for users. For
the evaluation of PDF documents linked within the pages. In instance, out of the 730.116 PDF files evaluated, only 14.207
our study, of the 24.576 websites evaluated, 20.180 websites PDFs (1.94%) were free of accessibility errors, revealing
(82.1%) contained at least one linked PDF document. This how pervasive accessibility issues are in document-based
substantial proportion suggests that a significant number of content.
websites extend beyond HTML content and include down- Given that PDF documents are often used for critical con-
loadable PDF files. The total of 730.116 PDF documents tent such as forms, policies, and reports, their accessibility

Fig. 9  Success and failure in

accessibility validation process

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Universal Access in the Information Society

is as crucial as that of the web pages themselves. Therefore, monitoring was done in Nov.–Dec. 2023. As you can see,
validating both web pages and PDF documents provides a PDF18 is the techniques with more errors, it involves the
more accurate representation of a website's overall accessi- document title and specifies that it should be visible; the
bility status, ensuring that all types of content are inclusive following technique is the PDF9 checking the existing of
and usable for individuals with disabilities. heading tags; the third is MPDF1, technique of MAUVE
Table 1 indicates for each region, the number of web sites Best Practices which verifies the use of tags to define a
validated, the number of PDF documents validated, and the logical structure of documents; PDF16 checks whether
average accessibility for the Web, the PDF content, and their the Language attribute has been set within the document
combined accessibility. In general, it emerges that the PDF properties; the last one is the technique MPDF2, which
content is less accessible than the Web one. checks whether the PDF is composed of a scanned image.
Table 2 shows the list of the techniques which returned The results have been provided to the national accessibility
the most failures; note that not all the techniques described authority, which uses them to identify the public organisa-
in this paper were implemented when this large-scale tions to stimulate to improve the accessibility of the digital
content that they provide.

Table 1  MAUVE++ Metrics Region # websites # PDF docs Web accessib Web PDF accessib Com-
grouped by Italian regions complete- bined
ness accessib

Lombardia 3953 3060 78 78 34 56

Lazio 2251 1686 78 78 34 56
Campania 1910 1568 80 79 34 57
Piemonte 2348 1522 80 77 30 55
Sicilia 1745 1467 78 77 32 55
Veneto 1709 1227 79 78 30 54
Emilia-Romagna 1526 1121 79 78 32 55
Toscana 1320 1022 79 79 31 55
Puglia 1248 998 79 79 34 56
Calabria 862 705 77 79 35 56
Trentino-Alto Adige 940 686 75 80 32 53
Sardegna 843 635 79 79 32 55
Abruzzo 720 586 79 79 30 54
Friuli- Venezia Giulia 645 541 78 74 31 54
Marche 661 520 75 79 32 53
Liguria 624 437 80 78 30 55
Basilicata 314 263 79 78 31 55
Umbria 344 256 77 78 30 53
Molise 237 214 77 78 30 53
V. d'Aosta 177 113 76 78 29 52
Total/Average 24,377 18,627 78 78 32 55

Table 2  PDF techniques failed Techniques Tech description Violation count Viola-
in large-scale evaluation tion
percent

PDF18 Specifying the document title using the Title entry 678,432 93
PDF9 Providing headings to mark content in PDF 662,143 91
MPDF1 Use of tags to specify the logical structure of PDF 425,436 58
PDF16 Setting the default language with/Lang entry 392,065 54
MPDF2 Searchable text in PDF document 107,851 15

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 Universal Access in the Information Society

6 User test acknowledged that the participation was voluntary and it

could cease at any time, and their privacy would have been
We carried out a user test to understand the usability of safeguarded. The tests were performed remotely with the
the tool’s new features, by collecting not only information support of a videoconference system (users were allowed
about how the tool was subjectively perceived by users, using their preferred one), and the users were asked to
thanks to qualitative user observations, but also more share their screen with a moderator, during the whole test.
objective usability-related data (task time, task success, The duration of each test session (from the moment users
SUS questionnaire). started to complete the questionnaire until they submitted
it) ranged between 22 and 53 min, with an average value of
33 min. Users did not receive any compensation for their
6.1 Participants and test organisation test participation.

We involved a new group of 18 participants (11 males, 6.2 Tasks and questionnaires

4 females), with ages ranging between 40 and 62 years
(Mean = 48.4, Std. Dev. = 8,5). As for the recruitment, The tasks of the test were identified to let the users try the
most users (14 out of 19) belonged to organisations who functionalities recently introduced for a combined analysis
participated in a webinar organised under the aegis of an and validation of Web and PDF content, and then, by gather-
Italian National Project related to monitoring accessibil- ing useful data during the execution of such tasks (e.g., time
ity in public administrations in Italy. The other partici- on task), allowing us to derive hints about their usability.
pants involved in the test were either colleagues of them, First, we started with a group of tasks associated with the
or people working in public organisations and involved in validation of a single web page together with its linked PDF
website management. files (Requirement R2).
Participants first completed a socio-demographic pre-
test questionnaire, to gather information about them and • Task 1: Carry out a single page validation on a specific
their background. One user declared having a physical/ page, then identify the WCAG Success Criterion most
motor disability (of the legs), which anyhow did not pre- often violated and the number of times it was violated;
vent him from participating in the test. Another user had • Task 2: Analyse the results obtained by verifying the
a hearing disability and then, for the call, he asked to accessibility of the first PDF document identified by
use Microsoft Teams, since it supports live captioning. the tool as associated with the considered page: (i)
Regarding the role they play in their organisation (it was provide the URL of this PDF, (ii) report the number of
possible to indicate more than one), 12 users self-identi- errors identified in it by the tool, and (iii) how many of
fied them as accessibility experts, 4 as Web developers, 2 them could be fixed by it; (Note: the URL of the PDF
as Web commissioners, 2 as content providers, 2 indicated was requested to the participants to be sure that all the
other roles (e.g., managing specific services within pub- reported metrics referred to the same PDF);
lic administrations). In terms of familiarity with tools for • Task 3: Consider all the PDF files linked to the consid-
checking web accessibility, two users declared not having ered page and report the total number of errors identified;
used any of them, while the others had already had some • Task 4: Provide the combined accessibility percentage
experience. Among the tools mentioned by them were: associated with the page.
previous versions of MAUVE++ (6 users), SiteImprove (5
users), Wave (3 users), Accessibility Insights, Lighthouse, Afterwards, we asked the participants the following ques-
tools indicated by the W3C (3 users), as well as Chrome tion: “Did you find any difficulty in carrying out these tasks
extensions, Accessiway, Axe, Equalweb, aria-devtool (all or in understanding any of the associated information? If so,
of the latter ones were mentioned by one user). In addition, could you please provide further details or any suggestions
6 users declared having used MAUVE++ just a few times for improvements?”.
before the test; 2 users did not know about the tool before Then, another group of tasks related to the validation of
it; 4 users knew about the tool but had not used it before; multiple web pages and their linked PDF files, also consid-
three users had used the tool several times. ering multiple audits over time (Requirements R3, R4). To
Before the test, users received an email briefly introduc- this end, users had to consider a MAUVE++ project created
ing test goals, together with a short presentation providing before the test, and to be used for monitoring the accessibil-
some information about the new features of the tool. Users ity of a set of pages across multiple audits:
also received an informed consent document to fill in and
sign before the test: this document, among other things, • Task 5: Report the total number of PDF files that the tool
identifies as linked to the pages included in the project;

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Universal Access in the Information Society

• Task 6: Among the available audits of the project created 6.3 Results

for the test, identify the audit in which the PDF acces-
sibility percentage is best; 6.3.1 Task time
• Task 7: Analyse the information associated with the pro-
ject's most recent audit and indicate the web page (or web Since accessibility validation can be a rather long and tedi-
pages) having the most errors; ous activity, we considered the time (in seconds) taken to
• Task 8: Still considering the most recent project audit, carry out the various tasks as a relevant metric to under-
indicate the PDF document with the highest error num- stand the usability of the tool, since the shorter the time on
ber; a task, the more efficient (and thereby usable) the support-
ing features could be. Task time was measured from the
Finally, we asked users to perform the validation of a PDF time the user finished reading (aloud) the task instructions,
file and semi-automatically correct the errors that the tool to the time they finished the task (i.e., when they found the
could fix (Task 9, see Requirements R1, R5). required answer or completed the task actions).
After the tasks were completed, participants were asked Figure 10 reports the time needed to carry out the tasks:
to respond to some questions: users spent longer times on Task 9, Task 1 and Task 6.
Task 9 was the longest one because it consisted of multiple
• Did you encounter any difficulties in performing this steps done one after the other: users had to validate a PDF
task or in understanding the information/functionality file, make the tool fix the errors it is able to correct, and
provided by the tool? If so, what did you find unclear or validate again the corrected file. As for Task 1, first-time
unclear, and why? users spent some more time on this task compared to users
• Overall, do you think the tool provides useful informa- more familiar with the tool because the presentation of the
tion/functionality for integrated web + pdf accessibility errors in terms of success criteria required first selecting
analysis? Is there any other information that you think a specific tool view (End-user) and then select a specific
would be useful to add or modify to improve the inte- sub-panel (“Success Criteria”): this was not intuitive for
grated analysis (web and pdf)? them. In Task 6 most users spent more time than expected
• Overall, do you think the tool provides this information because they first looked for the required information
clearly? Is there anything you would change/add/delete to within each audit (for manually comparing the values of
improve the integrated analytics (web and pdf) supported different audits with each other), whereas the tool already
in the tool? provides this info at the project level (as it refers to multi-
• Based on your experience in this test, what are the three ple audits). Unfortunately, not many participants quickly
things about the tool that you liked the most? noticed this feature.
• Which three aspects did you like the least?
• Do you have any further suggestions for improvements?

Fig. 10  Average task time (in

seconds)

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 Universal Access in the Information Society

6.3.2 Task success (slightly wrong answer, so it was classified as a minor

problem).
We categorised the performance of users on the various Task 4 Thirteen users answered this question correctly.
tasks in terms of these levels: (complete) success, minor The five users who answered incorrectly seem to report dif-
problems, major problems, failure. Figure 11 shows a ferent metrics (i.e., the PDF accessibility percentage), thus
graphical summary showing the task success levels, cal- their task outcomes were classified as “Failure”. This may
culated per task. suggest that especially metrics that are expressed using the
Task 1 Fourteen users correctly identified the most often same measure unit (e.g., in percentage terms) could be a bit
violated success criterion. Of these, thirteen users cor- confusing for some users and therefore they should be better
rectly identified the number of times it was violated as differentiated in the tool.
well, while one user reported a wrong value (thereby, the User Comments regarding Task 1—Task 4 After carrying
success level for this task was minor problem). Among out Tasks 1 to 4, users could report how difficult they per-
the four that did not provide the correct answer, the most ceived them to be. Twelve users declared they had been easy.
common error was to report the most violated technique One found some difficulties in Task 3: he was “expecting to
and not the most violated success criterion: three users find in some point of the tool the info about the total number
made this error (e.g., one reported Tech G18, violated 20 of errors associated with all the PDF files analysed”. This
times): all such cases were considered major problems. information was available in the tool, but that user missed
The remaining user reported the level of conformance it. One user did not immediately find the button supporting
(“Level AA”, violated 3 times), and this was classified as the validation of the PDFs linked to a web page. Two users
“Failure”. Thus, users who made errors in this task seem found that the web-based report might not be always clear
to have specific difficulties more at a conceptual level (i.e., in all its details, one of them found the PDF version clearer.
the difference between WCAG success criteria and tech- One user had difficulty separating the results that refer only
niques does not seem clear to them). to PDF files from the others. Another found that the tool
Task 2 All participants gave the correct answers to all had too many tabbed panels: she suggested having a menu
three questions except two: one identified only one error instead.
to be fixed by the tool, instead of two (thus, this was clas- Task 5 All users except two gave the correct answer (6
sified as a minor problem), another user did not identify PDF files), while two users identified a different number of
the right PDF file for the analysis. PDF files (major problems). Very few users answered this
Task 3 Thirteen users gave the correct answer to this question after manually calculating the requested value (as a
question (the tool identified 5 errors), thereby correctly sum of the number of PDF files linked to the pages included
fulfilling this task; the remaining five answered that in the project). This was because, after opening the project
the tool identified a slightly different number of errors page, they quickly selected a specific audit of the project

Fig. 11  Success levels, calcu-

lated per task

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Universal Access in the Information Society

and then analysed the information associated with that audit, would also include a summary of the projects on the page
only discovering after a while that the requested information presented just after login”. Another user suggested: “There
(the total number of PDF files) was provided in the panel is a lot of useful information, and you could get lost. Perhaps
displaying information about the project. This choice was some info as soon as you hover the mouse over one tab or
made because this info does not change from one audit to one functionality could help use the tool better”. Another
the next, as the PDF files (and web pages) belonging to a user would like to have fewer limitations on the maximum
project are collected by the crawler before starting the actual number of pages to consider for analysis (currently set to
validation. Even though these users analysed the information 50). Another user suggested that it could be useful to have
provided in each single audit (rather than analysing the sum- personalised reports of the result of the validation.
mary info provided at the project level), this did not prevent Clarity of the Tool According to users, the tool clearly
them from successfully carrying out this task. indicates the information and ‘guides’ the users towards it.
Task 6 Fifteen users provided the correct answer. Of the One user suggested including a guide/tutorial of the new
remaining three, one provided only a number (30) as the features within the tool. Another suggested providing some
answer, while the other ones gave another URL (e.g., the audit-related metrics in a clearer and more visible manner. A
URL of the project homepage). These incorrect answers user suggested slightly modifying the way some information
(classified as major problems) highlight that these users is presented in the tool: tabs, in her opinion, are not able to
misunderstood what was requested, emphasising that some quickly capture users’ attention. Still regarding tabbed pan-
familiarity with the terms used in the tool is necessary to els, another user said, “Perhaps some info when you hover
correctly use it. While almost all users successfully per- the mouse over one tab or functionality could help you use
formed this task, nonetheless, it was the one that took the the tool, and perhaps a side menu that allows you to navi-
most time. gate in a more streamlined way, in the sense that you have
Task 7 All users were successful with this task. all the page options visible”. Another user said: “The info
Task 8 All users except one (who did not provide the provided seems clear and complete to me. I haven't found a
URL of a PDF file, so major problem) correctly carried out way to download the reports, if missing it might be useful to
this task. add it.”. One user suggested being more transparent on the
After carrying out Task 5 to Task 8 users were asked kind of modification done when using the ‘Quick fix’ func-
whether they had any difficulty with them. Two users did tionality, while another user said that it would be better to
not mention any difficulty, while all the remaining users clarify whether the tool validates all the checkpoints stated
mentioned not being able to immediately find the part of as supported by the tool, as in validation results not all the
the tool reporting the metrics associated with the different checkpoints are listed.
audits. This info is shown in a part of the tool UI that is not The tool’s three most appreciated aspects Five users
immediately visible when the user opens the project page, mostly appreciated that the tool delivers the results quickly.
but it is visible only after scrolling a bit. One user reported Six users reported the possibility of fixing errors. Five users
not to have understood at the beginning how to use the infor- said that the tool is easy/simple to use. Three users appreci-
mation associated with different audits of the same website. ated its clarity. Two users like its usability. Other aspects
Task 9 All users successfully carried it out, except one mentioned (each mentioned by just one user): the possibil-
who experienced a potential small bug, and another one who ity to easily find the Success Criteria; no need to install any
did not specify the title of the pdf file to fix. Most users application, to carry out a combined analysis of web pages
declared not having found any difficulty in carrying out this and PDF files, monitor projects over time, and validate more
task. One suggested slightly modifying the name of the cor- than one page; the use of icons for identifying errors; the use
rected file to distinguish it from the original one. of intuitive percentages; the possibility to have summaries
with the results associated with PDF file, the indication of
6.3.3 User observations and SUS questionnaire the frequencies of the most recurring errors, the possibility
to know the number of line where an error is located.
We also asked further questions to users. Below we sum- The tool’s three least appreciated aspects Aspects men-
marise their answers. tioned: the tool is available only in English (2 users); dif-
Usefulness of the Tool All users gave positive feedback on ficulty with logging in the tool (1 user, who added that it is
this aspect. Suggestions regarded providing a brief presenta- not clear when the tool is presenting the results associated
tion/tutorial of the new features of the tool, directly within it. with the web part, the PDF-related part, or the combined
One user suggested providing a summary of the combined one); threshold set on the maximum number of web pages
analysis also considering PDF files, by using e.g., a table. the tool currently allows to analyse in a validation (1 user);
Another user provided a similar comment: “I would find it the graphics of the tool could be improved (3 users); the use
interesting to be able to download the PDF results report. I of tabbed panels (“tabs don't capture much my attention”: 1

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 Universal Access in the Information Society

user); the reports provided by the tool (2 users found them improve. As for Task 1 (carry out the validation of a single
not readable, one of them said that it is because “this test web page and identify the WCAG success criterion most
is not included into a real situation”; some usability issues, violated and how many times it was violated), although it
both in terms of navigation (2 users) and of visibility of might represent a typical accessibility evaluation task, in
buttons (1 user); some metrics associated with audits are this case several problems were associated with a misinter-
not very visible (1 user); the responsiveness of the tool (two pretation of the task outcomes (e.g., some users reported
users) One user said that it was a bit confusing, especially at the most violated WCAG technique rather than the most
the beginning, distinguishing between the errors associated violated WCAG success criterion, as requested by the task).
with web pages and those associated with PDF files. This issue, while stressing on the importance of providing
Further Suggestions Two users suggested simplifying users of accessibility evaluation tools with useful resources
the navigation, as the tool provides a lot of info, currently to improve their understanding of WCAG standards, or bet-
(e.g. using a menu or a breadcrumb). One suggested provid- ter highlighting concepts that might create potential con-
ing a (video) tutorial for explaining the new features of the fusion among them, highlighted an issue in understanding
tool. One suggested increasing the max number of pages to the structure of the WCAG guidelines and in particular the
consider in a single audit. Four users declared not to have differences between success criteria and techniques and the
anything to report on this aspect. articulated relationships between them. Moreover, for Task
We also submitted a System Usability Scale (SUS) ques- 6 (identify the audit having the best accessibility percent-
tionnaire to users. The resulting average SUS score [11] was age of PDF files), again, most of the problems in the test
good, with value = 76.5 (min = 45, max = 95, st. dev. = 14). were found in users misinterpreting the output of the task
requested, probably due to a lack of user familiarity with the
tool and some related (general and specific) concepts. Thus,
7 Discussion for all such cases in which the success rates were not par-
ticularly high, we expect that, as users became more familiar
In this section, we aim to derive more general outcomes and with concepts and metrics connected with the tool (and even
present implications from the results of this work. The tool with more general WCAG-related concepts), their ability
has demonstrated its scalability by completing a large-scale to leverage on the information provided by it could likely
accessibility validation in which it successfully evaluated improve.
4.235.456 web pages and 730.116 PDF documents. The The high number of users who successfully carried out
high number of web pages directly linked with PDF docu- Task 7 (100% of users identified the web pages having the
ments demonstrates how these two digital worlds are tightly most errors) and Task 8 (94.4% of users identified the PDF
connected, and thus an integrated accessibility validation file with the highest number of errors within a specific
is necessary for supporting all users. The high number of audit), highlight that, for tasks involving typical objectives
accessibility violations found in the PDF documents ana- of accessibility verification (e.g., identifying the parts of a
lysed indicates that there is a need to better stimulate the web site having the most errors), the tool seems to be very
PDF authors to pay attention to accessibility issues. effective in supporting users.
Looking at the data of the user test, if we analyse the data To sum up, the tool apparently was more effective in sup-
obtained for task success levels, calculated per task, we can porting users while doing tasks involving concepts and activ-
note that some problems were associated with the execution ities which users should be already familiar with, (e.g., as
of Task 4 (identify the combined accessibility percentage they already know what to look for using the tool), whereas
associated with a web page), followed by Task 1 (carry out more difficulties occurred when tasks involved handling new
the validation of a single web page and identify the WCAG concepts/metrics, or a more in-depth knowledge of WCAG
success criterion most violated and how many times it was related concepts (e.g., the relationship between WCAG suc-
violated) and then Task 6 (identify the audit having the best cess criterions and WCAG techniques). As a design implica-
accessibility percentage of PDF files). As for Task 4, the tion, especially when tools introduce new concepts/metrics,
metrics of ‘combined accessibility percentage’ required by proper explanations of them should be communicated/con-
this task may have represented a concept a bit unfamiliar veyed in the tool, which would be particularly beneficial to
to users, since it is not usually supported by accessibility unfamiliar (or less skilled) users.
tools. Thus, users may have experienced some difficulties As for task time, the tasks that were identified in the user
in correctly interpreting that value. We expect that, as users study were aimed at mirroring typical ‘scenarios’ that acces-
became more accustomed to that metric/concept and its sibility experts might want to carry out with our tool, sup-
potential benefits (quickly gain a ‘summary’ value of the porting automatic accessibility verification of web pages and
accessibility of a web page, also considering the associated associated PDF files. We focused on the most novel features
PDF files), their ability to leverage its value could likely that our tool supports for the accessibility evaluation of a

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Universal Access in the Information Society

combination of web pages and PDF files. Indeed, as men- considerable amount of time compared to the use of manual
tioned, with the tasks identified in the user study we aimed methods, especially for repetitive tasks such as identifying
to consider mainly 3 scenarios: Task 1 to Task 4 supporting errors at audit/website level, at single page level, and at the
the validation of a single web page and its associated PDF level of a single PDF file (although manual verifications
files, Task 5 to Task 8 aimed at supporting the evaluation should always be carried out to provide a comprehensive
of accessibility of a group of pages (including the associ- evaluation of accessibility).
ated PDF files) over time, whereas Task 9 supporting the
verification of accessibility of a PDF file, possibly automati-
cally correcting some of its errors. Given the logical coher- 8 Conclusions and future work
ence that those scenarios have, we can try to discuss the
results obtained in the user study, as they can be more easily In this paper, we present a novel solution to support a com-
referred to more general (tool-independent) activities that bined accessibility analysis and validation of web pages,
accessibility experts might want to conduct. and the PDF files linked to them, which has been obtained
According to this, we can see the sum of the times associ- by extending a tool for Web accessibility validation. We
ated with the execution of the first group of Tasks (1, 2, 3, describe the functionalities it supports, the results obtained
4) as the time required, on average, to carry out tasks useful in a large-scale validation carried out on Italian public ser-
to evaluate the accessibility of a web page and the related vice websites, and the feedback received in a usability test.
PDF files, more specifically: (1) validate a single web page While the test highlighted some points to improve from a
together with its linked PDF files; (2) identify the WCAG usability perspective, the tool was overall perceived as useful
Success criterion most violated and how many times it was and intuitive by users. The tool has also shown its usefulness
violated; (3) identify the number of errors found in a spe- and suitability in being exploited for large-scale monitoring,
cific PDF file linked to that page, as well as the total num- thereby showing its ability to support comprehensive acces-
ber of errors found considering all the PDF files linked to sibility evaluation while handling large numbers of pages.
that page; (4) obtain the combined accessibility percentage In this way, it provides useful information for accessibility
associated with the concerned web page. If we sum these stakeholders to identify the areas that need more immediate
values, we obtain 222.6 s in total (~ 4 min). The Tasks of interventions.
the second group (5, 6, 7, 8) are focused on the validation In future work, a large-scale combined evaluation of both
of multiple webpages and PDF documents, namely to: (5) web pages and linked PDF documents based on an inter-
identify the PDF files linked to the concerned pages; (6) national sample of website URLs (i.e., Tranco list2) will
identify the audit in which the accessibility evaluation of be considered to extend the analysis beyond Italian public
PDF files was the best; (7) identify the web page having administration websites, broadening the scope of accessi-
the highest number of errors; (8) identify the PDF file with bility analysis. Additionally, we plan to extend the set of
the highest number of errors. If we do the same sum of the supported PDF techniques and to support newly introduced
completion time for the second group of Tasks (5, 6, 7, 8), success criteria in WCAG 2.2. We also aim to explore the
we obtain 246.6 s in total (~ 4 min), which correspond to the use of Large Language Models to further support the acces-
same time needed for doing an accessibility evaluation of sibility validation activity.
just one webpage with the relatives PDF documents (Tasks
Acknowledgements This work has been partially supported by the
1–4). Task 9 is the one corresponding to the validation of a Italian National Recovery and Resilience Plan (PNRR), Action 1.4.2.
PDF file and semi-automatically correct the errors that the
tool could fix (task time = 264 s, ~ 4 min). Thus, in total, Author contribution F.P. wrote the first three sections of the manu-
users involved in the test spent on average around 12 min to script. N.I. and M.M. wrote the “User Interface,” “Implementation,”
and “Application in National Monitoring Accessibility” sections. C.S.
get accessibility-related information (and even correct some wrote the “User Test” section. All authors reviewed the manuscript and
errors) of a specific PDF file, of a specific web page (and contributed to writing the conclusions.
its connected PDF files), and of a group of web pages (and
PDF files) over time. Funding Open access funding provided by ISTI - PISA within the
CRUI-CARE Agreement.
The complexity of the web pages and PDF files used in
the study, as well as the level of familiarity of the experts Data availability No datasets were generated or analysed during the
with the tool and the accessibility standards might affect current study.
their performance and thereby could have influenced such
results, which cannot be taken as general reference. None-
theless, these relatively short times for the supported tasks
(compared to the time that it would take to carry out the
same tasks manually) can confirm that the tool can save a 2
https://​tranco-​list.​eu/

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 Universal Access in the Information Society

Declarations 9. Iannuzzi, N., Manca, M., Paternò, F., Santoro, C.: Usability and
transparency in the design of a tool for automatic support for web
Competing interests The authors declare no competing interests. accessibility validation. Univ. Access Inform. Soc. 23(1), 435–454
(2022). https://​doi.​org/​10.​1007/​s10209-​022-​00948-x
10. Iannuzzi, N., Manca, M., Paternò, F., and Santoro, C.: Large
Open Access This article is licensed under a Creative Commons Attri- scale automatic web accessibility validation. In: Proceedings of
bution 4.0 International License, which permits use, sharing, adapta- the 2023 ACM Conference on Information Technology for Social
tion, distribution and reproduction in any medium or format, as long Good, September 06, 2023, pp. 307–314. ACM, Lisbon Portugal
as you give appropriate credit to the original author(s) and the source, (2023). https://​doi.​org/​10.​1145/​35825​15.​36095​49
provide a link to the Creative Commons licence, and indicate if changes 11. Jordan, P.W., Thomas, B., McClelland, I.L., Weerdmeester, B.
were made. The images or other third party material in this article are (eds.): Usability Evaluation In Industry. CRC Press, Boca Raton
included in the article’s Creative Commons licence, unless indicated (1996). https://​doi.​org/​10.​1201/​97814​98710​411
otherwise in a credit line to the material. If material is not included in 12. Martins, B., Duarte, C.: Large-scale study of web accessibility
the article’s Creative Commons licence and your intended use is not metrics. Univ. Access Inf. Soc. 23(1), 411–434 (2022). https://​
permitted by statutory regulation or exceeds the permitted use, you will doi.​org/​10.​1007/​s10209-​022-​00956-x
need to obtain permission directly from the copyright holder. To view a 13. Martins, B., Duarte, C.: A large-scale web accessibility analysis
copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. considering technology adoption. Univ. Access Inf. Soc. (2023).
https://​doi.​org/​10.​1007/​s10209-​023-​01010-0
14. Nganji, J.T.: The portable document format (PDF) accessibil-
ity practice of four journal publishers. Lib. Inf. Sci. Res. 37(3),
References 254–262 (2015). https://​doi.​org/​10.​1016/j.​lisr.​2015.​02.​002
15. Pandey, M. and Dong, T.: Blending accessibility in UI framework
1. Acosta-Vargas, P., Luján-Mora, S., and Acosta, T.: Accessibility documentation to build awareness. In: The 25th International
of portable document format in education repositories. In: Pro- ACM SIGACCESS Conference on Computers and Accessibility,
ceedings of the 2017 9th International Conference on Education October 22, 2023, pp. 1–12. ACM, New York (2023). https://​doi.​
Technology and Computers, December 20, 2017. ACM, Barce- org/​10.​1145/​35976​38.​36083​80
lona Spain. pp. 239–242 (2017). https://d​ oi.o​ rg/1​ 0.1​ 145/3​ 17553​ 6.​ 16. Pradhan, D., Rajput, T., Rajkumar, A.J., Lazar, J., Jain, R.,
31755​74 Morariu, V.I., Manjunatha, V.: Development and evaluation of a
2. Apache.: Apache PDFBox, A Java PDF Library. Retrieved from tool for assisting content creators in making PDF files more acces-
https://​pdfbox.​apache.​org/ sible. ACM Trans. Access. Comput. 15(1), 1–52 (2022). https://​
3. Broccia, G., Manca, M., Paternò, F., Pulina, F.: Flexible auto- doi.​org/​10.​1145/​35076​61
matic support for web accessibility validation. Proc. ACM Hum.- 17. Rajkumar, A.J., Lazar, J., Jordan, J.B., Darvishy, A., and Hutter,
Comput. Interact. 4(EICS), 1–24 (2020). https://​doi.​org/​10.​1145/​ H.P.: PDF accessibility of research papers: What tools are needed
33978​71 for assessment and remediation? (2020). https://d​ oi.o​ rg/1​ 0.2​ 4251/​
4. Carvalho, L.P., Guerreiro, T., Lawson, S., and Montague, K.: HICSS.​2020.​512
Towards real-time and large-scale web accessbility. In: The 25th 18. Ross, A.S., Zhang, X., Fogarty, J., Wobbrock, J.O.: An epidemi-
International ACM SIGACCESS Conference on Computers and ology-inspired large-scale analysis of android app accessibility.
Accessibility, October 22, 2023, pp. 1–9. ACM, New York (2023). ACM Trans. Access. Comput. 13(1), 1–36 (2020). https://d​ oi.o​ rg/​
https://​doi.​org/​10.​1145/​35976​38.​36084​03 10.​1145/​33487​97
5. Darvishy, A.: PDF accessibility: tools and challenges. In: Miesen- 19. Schiavone, A.G., Paternò, F.: An extensible environment for
berger, K., Kouroupetroglou, G. (eds.) Computers Helping People guideline-based accessibility evaluation of dynamic Web appli-
with Special Needs: 16th International Conference, ICCHP 2018, cations. Univ. Access Inf. Soc. 14(1), 111–132 (2015). https://d​ oi.​
Linz, Austria, July 11-13, 2018, Proceedings, Part I, pp. 113–116. org/​10.​1007/​s10209-​014-​0399-3
Springer, Cham (2018). https://d​ oi.o​ rg/1​ 0.1​ 007/9​ 78-3-3​ 19-9​ 4277- 20. Selenium.: Selenium Web Driver. Retrieved from https://​www.​
3_​20 selen​ium.​dev/
6. Debevc, M., Škraba, T., Cerovac, B., Kožuh, I., Rajh, N.: Monitor- 21. W3C.: Web Accessibility Evaluation Tools List. https://​www.​w3.​
ing website accessibility: evaluating current approaches and a pro- org/​WAI/​ER/​tools/ (2024). Accessed 9 Jan, 2024
posal for improvements. J. Accessib. Design All 13(2), 162–187
(2023). https://​doi.​org/​10.​17411/​jacces.​v13i2.​485 Publisher's Note Springer Nature remains neutral with regard to
7. EU Commission.: Directive (EU) 2016/2102 of the European Par- jurisdictional claims in published maps and institutional affiliations.
liament and of the Council. Retrieved from https://​eur-​lex.​europa.​
eu/​eli/​dir/​2016/​2102/​oj (2016)
8. Fernandes, K., Paramananthan, S., Cockburn, L., Nganji, J.: Read-
ily available but how accessible?: An analysis of the web accessi-
bility of healthcare-related resources. J. Access. Design All 13(2),
188–215 (2023). https://​doi.​org/​10.​17411/​jacces.​v13i2.​421

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Terms and Conditions
Springer Nature journal content, brought to you courtesy of Springer Nature Customer Service Center GmbH (“Springer Nature”).
Springer Nature supports a reasonable amount of sharing of research papers by authors, subscribers and authorised users (“Users”), for small-
scale personal, non-commercial use provided that all copyright, trade and service marks and other proprietary notices are maintained. By
accessing, sharing, receiving or otherwise using the Springer Nature journal content you agree to these terms of use (“Terms”). For these
purposes, Springer Nature considers academic use (by researchers and students) to be non-commercial.
These Terms are supplementary and will apply in addition to any applicable website terms and conditions, a relevant site licence or a personal
subscription. These Terms will prevail over any conflict or ambiguity with regards to the relevant terms, a site licence or a personal subscription
(to the extent of the conflict or ambiguity only). For Creative Commons-licensed articles, the terms of the Creative Commons license used will
apply.
We collect and use personal data to provide access to the Springer Nature journal content. We may also use these personal data internally within
ResearchGate and Springer Nature and as agreed share it, in an anonymised way, for purposes of tracking, analysis and reporting. We will not
otherwise disclose your personal data outside the ResearchGate or the Springer Nature group of companies unless we have your permission as
detailed in the Privacy Policy.
While Users may use the Springer Nature journal content for small scale, personal non-commercial use, it is important to note that Users may
not:

1. use such content for the purpose of providing other users with access on a regular or large scale basis or as a means to circumvent access
control;
2. use such content where to do so would be considered a criminal or statutory offence in any jurisdiction, or gives rise to civil liability, or is
otherwise unlawful;
3. falsely or misleadingly imply or suggest endorsement, approval , sponsorship, or association unless explicitly agreed to by Springer Nature in
writing;
4. use bots or other automated methods to access the content or redirect messages
5. override any security feature or exclusionary protocol; or
6. share the content in order to create substitute for Springer Nature products or services or a systematic database of Springer Nature journal
content.
In line with the restriction against commercial use, Springer Nature does not permit the creation of a product or service that creates revenue,
royalties, rent or income from our content or its inclusion as part of a paid for service or for other commercial gain. Springer Nature journal
content cannot be used for inter-library loans and librarians may not upload Springer Nature journal content on a large scale into their, or any
other, institutional repository.
These terms of use are reviewed regularly and may be amended at any time. Springer Nature is not obligated to publish any information or
content on this website and may remove it or features or functionality at our sole discretion, at any time with or without notice. Springer Nature
may revoke this licence to you at any time and remove access to any copies of the Springer Nature journal content which have been saved.
To the fullest extent permitted by law, Springer Nature makes no warranties, representations or guarantees to Users, either express or implied
with respect to the Springer nature journal content and all parties disclaim and waive any implied warranties or warranties imposed by law,
including merchantability or fitness for any particular purpose.
Please note that these rights do not automatically extend to content, data or other material published by Springer Nature that may be licensed
from third parties.
If you would like to use or distribute our Springer Nature journal content to a wider audience or on a regular basis or in any other manner not
expressly permitted by these Terms, please contact Springer Nature at

[email protected]