The current issue and full text archive of this journal is available on Emerald Insight at:

https://www.emerald.com/insight/0951-3574.htm

AAAJ
34,8 Sustainability rating and moral
fictionalism: opening the black box
of nonfinancial agencies
1740 Olivier Boiral
D
epartement de Management, Facult
e des sciences de l’administration,
Received 21 December 2019
Revised 27 June 2020
Universit
e Laval, Quebec, Canada
21 November 2020
Accepted 4 March 2021
David Talbot
ENAP - National School of Public Administration, Quebec, Canada
Marie-Christine Brotherton
D
epartement de Management,
Facult
e des sciences de l’administration, Universit
e Laval, Quebec, Canada, and
~aki Heras-Saizarbitoria
In
Department of Management, Faculty of Economics and Business,
University of the Basque Country, UPV∕EHU, San Sebastian, Spain

Abstract
Purpose – The purpose of this paper is to explore the practices, challenges and ethical issues underlying the
fabric and dissemination of corporate sustainability ratings.
Design/methodology/approach – Based on 36 semi-structured interviews with sustainability rating
practitioners, the study shows the trade-offs, ethical judgments and customizable aspects involved in rating
practices, which cannot rely only on formal and predefined methods.
Findings – In contrast with the official optimistic rhetoric about the rationality and rigor of sustainability
rating methods, agencies face serious challenges in the measurement and comparison of performance in this
area, particularly in terms of the aggregation of scattered and fuzzy indicators, commercial pressures and the
availability, materiality and reliability of the information collected. Despite these concerns, sustainability
ratings do appear to be useful in improving corporate responsiveness and increasing investor awareness of the
complex and difficult-to-measure aspects of nonfinancial reports.
Practical implications – Rating agencies should collaborate to set up common indicators that would be
easier for firms to produce and should better separate their sustainability rating production activities from
other services they offer to companies (e.g. consultancy).
Originality/value – This study contributes to the literature on the measurement and promotion of corporate
sustainability by analyzing rating practices through the lens of moral fictionalism, which here refers to the
human tendency to build ethical judgments on fictional but convenient and useful representations.
Keywords Corporate sustainability, Sustainability ratings, ESG, Ethical issues
Paper type Research paper

Introduction
Measuring and comparing sustainability performance has become an essential issue for
companies and stakeholders alike (e.g. Boiral and Henri, 2017; Chatterji et al., 2009; Igalens
and Gond, 2005; Mattingly and Berman, 2006; Parguel et al., 2011; Searcy and Elkhawas,
2012), and this issue has been extensively analyzed in the sustainability accounting and
reporting (SAR) literature (e.g. Cho et al., 2015; Cho et al., 2012b; Cho and Patten, 2007;
Merkl-Davies and Brennan, 2007; Moneva et al., 2006). The disclosure of information on
Accounting, Auditing &
Accountability Journal
environmental, social and governance (ESG) issues is driven by the search for legitimacy in
Vol. 34 No. 8, 2021
pp. 1740-1768
© Emerald Publishing Limited This study was funded by the Canada Research Chair on the Internalization of Sustainable Development
0951-3574
DOI 10.1108/AAAJ-12-2019-4356 and Organizational Responsibility.
the eyes of various stakeholders, including financial markets, which require increasingly Black box of
accurate and comparable data on corporate sustainability performance (Brown et al., 2009; Cho nonfinancial
et al., 2015; Deegan, 2010; Hahn and L€ ulfs, 2014). However, the reliability and transparency of
corporate ESG disclosures have been widely criticized in the literature (e.g. Boiral, 2016; Cho
agencies
et al., 2015; Hahn and K€ uhnen, 2013; Owen, 2006; Talbot and Boiral, 2015). Even if the
information disclosed is robust and of high quality, it does not necessarily allow companies to
be compared and ranked according to their ESG performance, which is necessary for socially
responsible investment (SRI) decisions. As a result, various organizations have developed a 1741
number of sustainability ranking systems, such as the World’s Most Sustainable Companies,
the Corporate Sustainability Index, the 100 Best Corporate Citizens and the Corporate Human
Rights Benchmark. Sustainability rating agencies (SRAs) – sometimes referred to as
nonfinancial, extra-financial, ESG or social rating agencies – have become the main actors in
the promotion of ESG or sustainability ratings (SRs) (Chelli and Gendron, 2013; D
ejean et al.,
2004; Scalet and Kelly, 2010; Van Den Brink and van Der Woerd, 2004). As an intermediary
between companies and investors, SRAs act as information brokers by organizing,
transforming and redistributing information from various sources. In doing so, they bring
meaning and order to complex and difficult-to-measure sustainability issues (Escrig-Olmedo
et al., 2010, 2014; Kotsantonis et al., 2016; Rahdari and Rostamy, 2015; Windolph, 2011).
Although SRAs exert an increasing disciplinary power over organizations and financial
markets (Chelli and Gendron, 2013; Sauder and Espeland, 2009), their internal practices
remain opaque and have not been the object of in-depth studies (Chatterji et al., 2009; Chelli
and Gendron, 2013; Saadaoui and Soobaroyen, 2018; Windolph, 2011). Moreover, while the
validity and meaning of SRs have been questioned in the literature (Chatterji et al., 2016;
Delmas and Blass, 2010; Dorfleitner et al., 2015; Semenova and Hassel, 2015), many
stakeholders, including much of the SRI industry, assume these rankings accurately and
reliably measure and compare sustainability performance. This reflects Vaihinger’s
philosophy of “as if” (2014), by which humans accept fictions to give order to a chaotic world.
The objective of this paper is to explore the practices, challenges and ethical issues
underlying the fabric and dissemination of SRs based on the perceptions of practitioners
involved in this field, including SRA analysts. This paper addresses four important issues
that remain overlooked in the literature. First, the study opens the “black box” of SRAs
(Chatterji et al., 2009; Chelli and Gendron, 2013; Saadaoui and Soobaroyen, 2018; Stubbs and
Rogers, 2013; Windolph, 2011) by providing insights from practitioners on the internal
practices, trade-offs and ethical judgments underlying the rating process. Although this
study is not intended to detail the formal methods used in SRs, which are mainly confidential,
the interviews cast more light on the challenges and pressures faced by SRAs. Second, the
paper contributes to the literature on the measurability of sustainability performance (Boiral
and Henri, 2017; Capelle-Blancard and Petit, 2017; Delmas and Blass, 2010; Escrig-Olmedo
et al., 2017; Gray and Milne, 2004) by exploring the difficulties faced in the fabric of SRs,
including in terms of information reliability, data aggregation, commensurability and the
integration of qualitative aspects. Third, the study shows that, in spite of the arguable
fictitiousness of SRs and the uncertainties surrounding them, paradoxically, SRs remain useful
tools to promote corporate sustainability among companies and investors. Fourth, the article
contributes to the SAR literature by providing a new, relevant and valuable theoretical
perspective to further analyze issues such as the challenges faced by different stakeholders for
measuring and comparing sustainability performance.
The remainder of the paper is organized as follows. First, the literature on SRAs is
summarized and analyzed through the lens of moral fictionalism, which helps to explain
the ethical and paradoxical aspects of SRs. Second, the qualitative methods used in this
study are described. Third, the data analysis is centered around three critical aspects: the
internal practices of SRAs, the challenges faced in the development of SRs and the role of
AAAJ SRs in promoting corporate social responsibility (CSR). Contributions to the literature,
34,8 managerial implications and avenues for future research are detailed in the discussion
section.

The institutionalization of sustainability rating agencies

The “black box” of sustainability rating practices
1742 The rapid growth of SRI partly explains the need to obtain reliable and comparable
information on corporate sustainability performance (Dillenburg et al., 2003; Diouf and Boiral,
2017; Koellner et al., 2005; Kotsantonis et al., 2016; Lackmann et al., 2012; Richardson, 2009).
SRI represents nearly 30% of management assets in the world, or more than US$22tn (Global
Sustainable Investment Alliance, 2016). Credible SRI decisions presuppose that reliable SRs
exist and that these can be easily used to select companies based on their ESG performance
(Escrig-Olmedo et al., 2010, 2014; Scalet and Kelly, 2010; Windolph, 2011). Although the
development of SRs remains one of the main activities of SRAs, these agencies may offer a
wide range of complementary services, including sustainability indices, ESG evaluation,
country assessment, nonfinancial research, risk assessment, proxy voting, controversy
monitoring, alert services, sustainability policy implementation, consulting and portfolio
analysis (Eccles and Stroehle, 2018; Escrig-Olmedo et al., 2014; Novethic Research, 2014;
Rutledge, 2015; Saadaoui and Soobaroyen, 2018; Stubbs and Rogers, 2013; Velez-Castrillon,
2014). The development of these often highly specialized services is shaped by commercial
objectives and differentiation strategies in an increasingly competitive market (Chelli and
Gendron, 2013; Saadaoui and Soobaroyen, 2018). While SRAs were initially relatively
heterogeneous and small, mergers and acquisitions have led to the concentration of rating
services with a few major players, including MSCI, Vigeo Eiris, ISS and Sustainalytics
(Avetisyan and Hockerts, 2017; Eccles and Stroehle, 2018; Novethic Research, 2014). The
growth of these agencies has made it possible to diversify their competencies, internationalize
their activities and improve their visibility among investors and companies (Novethic
Research, 2014). The institutionalization and recognition of nonfinancial rating practices
have been strengthened by the creation of certain standards, such as the Arista certification,
which aims to improve the professionalism of SRAs and the quality of the information
produced (Association for Responsible Investment Service, 2012).
Although the institutionalization of SRAs has significantly improved the recognition and
dissemination of SRs (Bessire and Onn
ee, 2010; Elbasha and Avetisyan, 2018; Sauder and
Espeland, 2009; Windolph, 2011), the methods, internal practices and difficulties of the rating
process remain poorly studied. In the absence of transparent information on their internal
operations, SRAs seem like “black boxes” (Bessire and Onn
ee, 2010; D
ejean et al., 2004;
Delmas and Blass, 2010). The literature on SRs mainly focuses on the institutional context of
these “black boxes,” the information employed and the implications of their outputs (e.g.
Adam and Shavit, 2008; Chatterji et al., 2016; Chelli and Gendron, 2013; Delmas and Blass,
2010; Parguel et al., 2011; Searcy and Elkhawas, 2012). Several studies have analyzed the
diverse origins of SRAs, the impacts of screening practices on the performance of SRIs, and
the added value of SRAs for financial markets, including risk reduction (Capelle-Blancard
and Monjon, 2014; Escrig-Olmedo et al., 2010; Fowler and Hope, 2007; Koellner et al., 2005;
Scalet and Kelly, 2010; Sodjahin et al., 2017). For example, the origins and objectives of SRAs
tend to shape their different conceptions of corporate sustainability and the approaches they
use to measure performance in this area (Eccles and Stroehle, 2018). Sch€afer (2005) identified
several rating schemes used by SRAs, including risk assessment, value enhancement,
industries of the future and quality management models. Yet the consolidation of SRAs, their
takeover by traditional financial players (e.g. Bloomberg, Morning Star), and the increasing
standardization of rating practices (Avetisyan and Gond, 2013) contribute to reducing the
diversity of these rating agencies. This consolidation also seems to be refocusing SRA Black box of
activities on more traditional financial objectives that may conflict with CSR values nonfinancial
(Avetisyan and Hockerts, 2017). Overall, the role of SRAs and their rating systems with
regard to the promotion of corporate sustainability remains highly controversial (Capelle-
agencies
Blancard and Petit, 2017; Chatterji et al., 2009; Chelli and Gendron, 2013; Delmas and Blass,
2010; Saadaoui and Soobaroyen, 2018; Stubbs and Rogers, 2013; Windolph, 2011). In a study
based on public information from SRAs, Chelli and Gendron (2013) show that SRs reflect a
numbers-based ideology characterized by a relatively narrow and arbitrary vision of 1743
corporate sustainability. These ratings are shaped by various linguistic strategies aimed at
enhancing the perceived rigor and legitimacy of agencies while minimizing or concealing the
more fundamental difficulties inherent in measuring sustainability performance. The
arbitrary nature of SRAs’ methods and outputs has also been highlighted by several studies
that demonstrate the lack of convergence of the rankings issued by different agencies
(Chatterji et al., 2016; Dorfleitner et al., 2015; Mooij, 2017; Semenova and Hassel, 2015).
Discrepancies between ratings should encourage stakeholders and researchers using these
rankings to be much more cautious about their validity as a proxy for sustainability
performance (Chatterji et al., 2016).
Although SRAs are reluctant to detail their rating methods (Chelli and Gendron, 2013;
Delmas and Blass, 2010; Windolph, 2011) and access to this information is consequently
limited, several studies have attempted to explore these methods. Saadaoui and Soobaroyen
(2018) have highlighted the similarities and differences in the rating methods used, including
in terms of the weighting of sustainability issues and screening criteria. These differences
may explain the lack of agreement between different SRs and seem to be partly related to
increasing competitive pressures and SRAs’ efforts to differentiate (Benijts, 2008; Saadaoui
and Soobaroyen, 2018; SustainAbility, 2018). Delmas and Blass (2010) have also illustrated
the methodological differences between SRAs, particularly in terms of emphasis on various
issues such as toxic emissions, regulatory compliance and the quality of sustainability
disclosure. Overall, despite SRA statements on their professionalism and methodological
rigor, SRs seem to be based on many trade-offs, particularly due to the lack of standardization
and transparency of the data collected (Chatterji and Levine, 2006; Delmas and Blass, 2010;
Graafland et al., 2004; Windolph, 2011). The lack of consistency in the aggregation of often
scattered, unclear and qualitative information has also been mentioned (Boiral and Henri,
2017; Capelle-Blancard and Petit, 2017; Chelli and Gendron, 2013; Igalens and Gond, 2005;
Windolph, 2011). To solve this problem, a few studies have proposed implementing fuzzy
logic methods to facilitate the aggregation of vague, ambiguous and heterogeneous data
(Escrig-Olmedo et al., 2014, 2017; Liern and P
erez-Gladish, 2018; Liu, 2007; Rajak and Vinodh,
2015). However, it remains unclear how SRAs perceive such methods and how they can be
implemented in practical terms to improve the validity and rigor of SRs. These uncertainties
about the feasibility of developing rigorous, comparable and commonly accepted SRs fuel
criticism regarding SRA’s opacity and lack of credibility (Capelle-Blancard and Petit, 2017;
Chelli and Gendron, 2013; Windolph, 2011).
Generally speaking, research on SRAs leads to apparently conflicting conclusions. On the
one hand, SRAs appear to be essential actors in supporting the rapid growth of SRI and
facilitating the comparison of corporate sustainability performance (D
ejean et al., 2004;
Escrig-Olmedo et al., 2010, 2014; Novethic Research, 2014; Rahdari and Rostamy, 2015;
SustainAbility, 2018). On the other hand, the difficulties inherent in measuring sustainability
performance and the opacity of the methods used raise questions about the credibility of SRA
outputs (Boiral and Henri, 2017; Capelle-Blancard and Petit, 2017; Chelli and Gendron, 2013;
Windolph, 2011). Although they seem contradictory, these two conclusions are not
necessarily mutually exclusive; indeed, they reflect the moral fictionalism underlying the
SRA industry.
AAAJ Sustainability ratings as moral fictions?
34,8 Moral fictionalism can be defined as moral claims or judgments based on convenient fictions
rather than objective descriptions of reality (Eklund, 2017; Joyce, 2001, 2005; Kalderon, 2005).
Although moral fictions are based on false, inaccurate or unsubstantiated statements, they
tend to be collectively accepted and can even be useful to promote the common good, prevent
misconduct or change individual behaviors (Nolan et al., 2005; Thomas, 2012). Moral
fictionalism stands in contrast to the reductionism of moral realism, which assumes the
1744 existence of objective and demonstrable ethical judgments or facts on what is right or wrong
(Kalderon, 2005; Kim, 2006; Nolan et al., 2005). From a fictionalist perspective, statements
describing phenomena should not be considered necessarily and unequivocally true but
should be seen instead as accepted and useful fictions (Nolan et al., 2005; Vaihinger, 2014). In
his seminal work on the philosophy of “as if,” Vaihinger (2014) shows many examples of the
human tendency to derive fictional and reassuring representations from the observation of
phenomena and to consider those representations as accurate and accepted descriptions of
reality thereafter. The moral fictionalist approach has been applied in many fields, including
mathematics, morality, sociology, history, linguistics and philosophy (e.g. Balaguer, 2009;
Fine, 1993; Nolan et al., 2005; Vaihinger, 2014), but it has only rarely been applied in
management literature. A few studies have used the moral fictionalism perspective to
describe the simplifying, narrative-based and self-referential nature of theoretical
developments disconnected from the description of actual facts (e.g. Grandy and Mills,
2004; Hambrick, 2007; Pfeffer, 2007; Phillips, 1995; Rasche and Behnam, 2009). For example,
Grandy and Mills (2004) criticize theories and models in strategic management for tending
toward hyperreality – that is, fictions based on discourses largely dissociated from the
complex reality of organizations. Although these theories appear disconnected from the
reality of organizations, they can be useful to give meaning to complex phenomena, to further
explore issues or for educational purposes. The fictionalist perspective has also been used to
describe the unsubstantial and narrative-based nature of organizational discourse (Brunsson,
1989; Santilli, 1984; Savage et al., 2018; Zbaracki, 1998). According to Savage et al. (2018),
organizational practices are embedded in fictions based on discourses and representations
often dissociated from reality, but they are nevertheless useful, particularly in transforming
intentions into actions: “organizations are fictions that are constructed through speech acts,
props and narration, based on more general acts of pretense or make-believe—for a purpose”
(p. 991).
In the same vein, the literature has shown that reassuring statements in corporate
sustainability are often fictitious and dissociated from actual practices and performance
(Boiral, 2013; Cho et al., 2015; Hahn and L€ ulfs, 2014). Neo-institutional studies have
demonstrated that this dissociation is shaped by impression management strategies and a
search for legitimacy, as organizations face increasing external pressures around
sustainability while often remaining reluctant to implement substantial changes (Boiral,
2016; Cho et al., 2012b; De Villiers and Van Staden, 2011; Deegan, 2010; Hahn and K€ uhnen,
2013). One can assume that similar statements apply to SRAs, whose reassuring but poorly
substantiated statements on the rationality and rigor of SRs often provide SRs with
legitimacy and offer comfort to various stakeholders, including rated organizations, SRI
practitioners and researchers using SRs as a proxy for corporate sustainability performance
(Chelli and Gendron, 2013; D
ejean et al., 2004; Gauthier and Wooldridge, 2018; Scalet and
Kelly, 2010). More specifically, one can assume that SR practices are shaped by moral
fictionalism, which has three main components: the existence of collectively accepted moral
or ethical judgments, the fictitious or unsubstantiated nature of such judgments and their
practical utility.
First, SR practices are based on making moral judgments on companies by distinguishing
between practices and activities that are ethically acceptable, or even virtuous, and those
considered contrary to prevailing ESG standards (Escrig-Olmedo et al., 2014; Hedesstr€om Black box of
et al., 2011; Mooij, 2017). This moral dimension has shaped the history of SRAs, which were nonfinancial
originally founded by religious congregations and social movements seeking to boycott
companies or states whose behavior they considered reprehensible (Bessire and Onn
ee, 2010;
agencies
D
ejean et al., 2013; Entine, 2003). Although SRAs have considerably diversified their
activities, most of their services are embedded in moral or ethical assessments of a company’s
activities. These assessments may be based on various approaches, including negative
screening (the exclusion of businesses involved in controversial sectors such as tobacco, 1745
weapons and alcohol), ESG analysis (the assessment of performance related to
environmental, social and governance issues), controversy analysis (the monitoring of
questionable practices through various sources) and norm-based analysis (the observation of
compliance with international conventions and standards).
Second, the fictitious character of SRs is reflected in their lack of convergence (Chatterji
et al., 2016; Dorfleitner et al., 2015; Semenova and Hassel, 2015), which suggests that
sustainability performance does not represent a clearly defined or definable reality that can
be unambiguously measured and compared. If one assumes that differences in rating
methods (Benijts, 2008; Delmas and Blass, 2010; Saadaoui and Soobaroyen, 2018;
SustainAbility, 2018) partly explain discrepancies in SRs, this may give more credence to
moral fictionalism, which posits that ethical judgment does not describe reality in itself, but
rather reflects convenient norms and arbitrary representation systems whose apparent
accuracy and objectivity is wrongly taken for granted (Joyce, 2001, 2005; Vaihinger, 2014).
Moreover, even assuming the existence of similar rating methods between SRAs, the
comparability of sustainability performance seems at best uncertain. Various studies have
highlighted the difficulty or even impossibility of credibly measuring such performance,
particularly due to the importance of qualitative issues (e.g. biodiversity conservation, human
rights), the complexity of the indicators used, the lack of information standardization and the
difficulty in aggregating scattered data on very different aspects (Boiral and Henri, 2017;
Capelle-Blancard and Petit, 2017; Escrig-Olmedo et al., 2017; Gray and Milne, 2004; Hubbard,
2009). For example, offsetting poor performance on one sustainability indicator with good
performance on another is a questionable practice (Boiral and Henri, 2017; Capelle-Blancard
and Petit, 2017; Windolph, 2011). The integration of a wide variety of noncomparable
indicators to produce a clean, composite and comparable rating also raises problems of
commensurability and oversimplification (B€ohringer and Jochem, 2007; Capelle-Blancard and
Petit, 2017; Escrig-Olmedo et al., 2017; Singh et al., 2012). Even assuming that fuzzy logic
approaches can partly address this problem (Escrig-Olmedo et al., 2017; Liu, 2007; Rajak and
Vinodh, 2015), the literature has consistently shown that there is a proliferation of unreliable
information on corporate sustainability, due to greenwashing tendencies and managerial
control over the disclosure process in particular (Boiral, 2013; Cho et al., 2015; Hahn and L€
ulfs,
2014; Moneva et al., 2006). Given the lack of reliable input data on corporate sustainability, it
seems unreasonable to assume that SRA outputs accurately reflect reality. More
fundamentally, the fictitious aspects of SRs reflect the vague and elusive definition of
sustainability, which has been widely debated in the literature (Boiral, 2013; Devinney, 2009;
Milne et al., 2006; Moneva et al., 2006). In the absence of clearly defined and circumscribed
realities to be measured, SRs can hardly be consistent from one SRA to another. These
difficulties challenge the predominance of approaches that assume sustainability
performance measurements are realistic, and to give more credence to postmodernist
approaches that emphasize the fictitiousness of the seemingly rational rhetoric on these
measurements. The discourse of realistic measurements “acts like a facade to provide an
organized, structured, rational, and controllable representation of fundamentally confused,
chaotic, and, above all, discursive issues” (Boiral and Henri, 2017, p. 292).
AAAJ Third, despite their fictitious aspects, SRs remain useful on at least three different levels.
34,8 First, the orderly, rigorous and rational appearance of SRs facilitates their use by various
stakeholders (Chelli and Gendron, 2013). Conversely, the information conveyed through
sustainability reports or other publicly available sources can hardly be directly used for
comparison purposes due to its complexity, lack of compliance and disordered appearance
(Boiral and Henri, 2017; Escrig-Olmedo et al., 2017; Friedman and Miles, 2001; Hubbard, 2009;
Talbot and Boiral, 2018). For example, in their study on Green House Gas (GHG) reporting in
1746 the energy sector, Talbot and Boiral (2018) showed that more than 95% of the information
disclosed in sustainable development reports – including those verified by external auditors –
did not comply with the GRI requirements that companies were supposed to apply and were
not comparable with one another. This type of noncompliance makes sustainability
disclosure very difficult to use, particularly for SRI managers who must constantly monitor
and compare the performance of a large number of firms. In this context, regardless of the
reliability of ratings, SRAs provide a sense of comfort and order with regard to information
that would otherwise be difficult, if not impossible, to rigorously compare. SRAs may
therefore facilitate the promotion of a complex and multifaceted concept through the
dissemination of SRs. Second, whatever the greenwashing tendencies of companies and
uncertainties surrounding SR methods, the dissemination of SRs may have a positive impact
on companies’ practices. Some studies on greenwashing and organizational performativity
suggest that sustainability statements may promote “aspirational talks” (Christensen et al.,
2013) conducive to more substantial changes, including changes to make the company more
consistent with the information it has disseminated (Bowen, 2014; Bowen and Aragon-Correa,
2014; Siano et al., 2017). In this perspective, SRAs could play a catalytic role by promoting
rankings that provide greater visibility to sustainability aspects that organizations have not
integrated well (Gauthier and Wooldridge, 2018). In addition, SRAs may use information not
controlled by companies and may further take into account the views of various stakeholders
on corporate sustainability (Diouf and Boiral, 2017; Kolk, 2004; Mu~ noz-Torres et al., 2019;
Perrini and Tencati, 2006). Third, regardless of SRs’ validity, the production of SRs can exert
a disciplinary function over organizations and investors alike. Various Foucauldian studies
have shown that monitoring encourages the internalization of certain standards and has a
disciplinary effect on organizations (Chelli and Gendron, 2013; D
ejean et al., 2004; Sauder and
Espeland, 2009). From this perspective, it can be assumed that SRA practices – including
requesting information from companies covered by SRs – exert some pressures on managers
to better integrate sustainability and to comply with SRA expectations.
Although moral fictionalism seems to shed new light on SR practices, the opacity of SRAs
and the lack of substantial empirical studies on this issue mean that SR practices require
further investigation. More specifically, the perceptions of SR practitioners have been
overlooked in the literature, with a few exceptions (D
ejean et al., 2004; Saadaoui and
Soobaroyen, 2018). Moreover, the few empirical studies on the subject have not really
investigated the moral fictionalism underlying SRA activities, in particular practitioners’
perceptions of the ethical aspects, meaning and usefulness of SRs.
The theoretical approach of moral fictionalism is a new, relevant and differentiated
perspective for shedding light on SRs as well as on the SAR field in general. Multiple
theoretical approaches have been proposed to analyze the raison d’^etre and the main practices
of the SAR field. Although, as pointed out by Gendron (2018), this research field seems too
complex and fragmented to be clearly categorized and delimited, we can nonetheless mention
a number of theoretical perspectives that have been focused on the other characteristics of
SAR beyond the functional ones, including legitimacy theory (e.g. Bebbington et al., 2008; Cho
et al., 2012b; Cho and Patten, 2007; Haniffa and Cooke, 2005; Rahaman et al., 2004), (neo)
institutional theory (e.g. Archel et al., 2011; Cho et al., 2015; Moneva et al., 2006), impression
management theory (e.g. Cho et al., 2012b; Diouf and Boiral, 2017; Merkl-Davies and Brennan,
2007; Merkl-Davies et al., 2011) and a diverse set of critical theories that could be included Black box of
under the umbrella concept of “functional criticism” (Larrinaga Gonz
alez, 2017). Legitimacy nonfinancial
theory aims to shed light on the drivers and practices of SAR considering that managers and
their organizations are motivated by social considerations, beyond the economic ones, such
agencies
organizations’ desire to ensure that society perceives them as obeying social rules (e.g.
Guthrie and Parker, 1989; Rahaman et al., 2004). The (neo)institutionalist approach focuses on
the processes of decoupling between formal elements, including SAR, which are essentially
driven by social and institutional pressures and internal practices (e.g. Boiral, 2013; Cho et al., 1747
2015; Heggen et al., 2018; Meyer and Rowan, 1977; Moneva et al., 2006; Roberts, 2018). What is
new about this approach is that it highlights the rational myths underlying SAR – i.e. its
symbolic nature, divorced from organizational sustainability and mostly intended to improve
legitimacy in the eyes of stakeholders (Boiral and Henri, 2017; Gibassier et al., 2018; Heggen
et al., 2018; Roberts, 2018) – and the reasons behind the logic of appropriateness and the social
construction of reality (Higgins and Larrinaga, 2014; Larrinaga Gonz
alez, 2017). Impression
management theories are employed in SAR to shed more light on how companies disclose
information they have manipulated, which reflects their opportunistic behaviors in the quest
for social legitimacy, image improvement and the desire to conceal poor performance (Cho
et al., 2012b; Diouf and Boiral, 2017; Perkiss et al., 2020; Solomon et al., 2013; Talbot and Boiral,
2015, 2018).
Although a set of works using the abovementioned perspectives do shed light on the
motivations behind reporting practices, their underlying strategies and their ambiguous
effects with regard to corporate sustainability, these approaches do not directly question
whether it is possible to report performance accurately through SAR. The moral fictionalism
perspective, which could be included under the umbrella of “functional criticism,” adds value
to the SAR literature, as it proposes a more fundamental and radical (i.e. focused on the roots)
reflection on the elusive nature of reality – in this case corporate sustainability performance –
and the inability of reporting tools to account for this elusiveness, irrespective of institutional
pressures and managers’ intentions. In other words, the symbolic and decoupled nature of
SAR does not simply arise from organizations’ concealment strategies, as claimed by neo-
institutional and impression management approaches. Rather, it also, more fundamentally,
arises from the impossibility of transparently capturing the fragmented, ambiguous and
context-dependent nature of most sustainability issues by using unidimensional indicators
(Boiral and Henri, 2017). To our knowledge, most critical approaches on SAR do not clearly
consider such impossibility. This includes approaches based on the decoupling between talk
and action, which assume that the divorce between reporting practices and sustainability
issues arises from organizational strategies and the managerial capture of the reporting
process (Crilly et al., 2016; Gibassier et al., 2018; Heggen et al., 2018; P
erez-L
opez et al., 2015;
Roberts, 2018). The same applies to the criticisms of the organizational hypocrite underlying
SAR, which do not question the possibility of substantive disclosure but rather focus on the
contradictions between talk and action on sustainability, which are assumed to move in
opposite directions (Cho et al., 2015).
Nevertheless, the absence of clear evidence or examples of SAR practices that effectively
describe the reality of organizational sustainability issues in a reliable, transparent and
verifiable manner should lead to a more fundamental questioning of the fictional and
narrative nature of these practices, independent of the strategies for legitimizing
organizations and managing social pressures that are widely studied in the literature. The
moral fictionalist approach does not presuppose an intentionality, a strategy or the existence
of institutional pressures behind the contradictions between talk and reality in SAR. On the
contrary, one can assume that SRAs are committed to providing as relevant and transparent
information as possible. However, more fundamentally, the reliable measurement of
sustainability performance simply appears to be an impossible quest. Another possible
AAAJ contribution of the fictionalist approach is to point out that, despite the fictitious character of
34,8 SAR practices, they may paradoxically be useful or even necessary. As the seminal work of
Vaihinger (2014) on the philosophy of “as if” points out, while the fictions developed to
account for reality are false and misleading, they often have a practical use in constructing a
representation of the world and acting on it. Fictionalism thus offers a new perspective for
understanding SRs by taking into account both their character as symbolic, fictional and of a
discursive nature, and their relevance for indirectly changing the practices of the
1748 organizations being rated.

Methods
The objective of this study was to shed more light on the fabric of SRs and their dissemination
by examining the perceptions of SR practitioners with regard to the practices, challenges and
ethical implications of the rating process. This empirical study was therefore initially not
intended to validate the moral fictionalist approach or to identify elements related to this
theory; its relevance emerged later. The exploratory nature of this study and its focus on the
respondents’ perceptions required a qualitative approach, which is best suited to exploring
the meanings behind under-researched and complex issues (Corbin and Strauss, 1990;
Gephart, 2004). Semi-structured interviews were performed to collect perceptions of SR
practitioners involved in the rating process. The grounded theory approach was then used; it
is commonly used in qualitative studies to develop theories and explanations derived from
the data rather than to validate pre-established hypotheses (Martin and Turner, 1986; Strauss
and Corbin, 1994; Suddaby, 2006). For the present study, the theoretical perspective of moral
fictionalism was chosen and the literature review was written after the data collection and
analysis steps were performed, as suggested in the literature on grounded theory and the
inductive approach (e.g. Dunne, 2011; Thomas, 2006).

Data collection
The data collection was carried out in two stages. First, six nonfinancial rating agencies
with activities in North America were contacted. The majority of the agencies contacted
agreed to share some internal documents in order to make it possible to evaluate their rating
processes. In addition, interviews were conducted (N 5 6) with SR practitioners in order to
gain a better understanding of the issues and challenges of SRAs. The objective of this first
step was to better understand the rating processes, methodologies and challenges faced by
agencies.
Second, in order to deepen our understanding of some of the issues raised and of SR
practices, a second series of interviews was conducted. As the research team included a
former SRA employee with many contacts in this industry, participants were mainly
recruited through the snowball sampling method (Miles et al., 2018). Professional websites,
such as SRI-Connect and LinkedIn, were also used to facilitate the recruitment process.
At this stage, 300 people were contacted, either by email (87) or through these professional
websites (213). Of this group of 300 people, 30 agreed to participate in an interview. The low
response rate (10%) might be explained by the sensitive nature of the SR industry, as some
SRAs refused to let their employees participate in our study on the basis of confidentiality
and business interest. The anonymity of the participants was ensured by a thorough research
protocol approved by the university ethics committee prior to the data collection process.
Each participant had to sign a consent form ensuring confidentiality prior to the interview.
In total, 36 people participated in the interview process (6 in the first stage, 30 in the second)
(see Appendix 1). Interviews were conducted between November 2015 and June 2018 and
lasted between 45 and 90 min each. They were mainly conducted by phone or by Skype,
though a few interviews (4) were conducted face-to-face. Previous studies (e.g. Holt, Black box of
2010; Stephens, 2007; Sturges and Hanrahan, 2004) have shown no significant differences nonfinancial
between phone or Skype interviews and face-to-face interviews. All interviews were
transcribed verbatim using Microsoft Word. Given the sensitive nature of the SR process,
agencies
the interviews were focused on the respondents’ perceptions. It was impossible to gather
detailed information about SRA methodologies or outputs, aside from the documents
obtained in the first step of the study. The semi-structured interview guide revolved around
seven issues: the respondent’s background, the rating process, relationships with the rated 1749
companies, the reliability of the rating process and of the information collected, the concept of
materiality, standards and certifications, and various shareholder engagement processes.

Data analysis
The grounded theory approach involved the coding and grouping of passages under similar
themes following an inductive and iterative process grounded in the data. First, the
transcripts of the interviews were transferred to QDA Miner (v.4.1.21), a qualitative analysis
software used to help sort, analyze and retrieve information related to various categories.
Overall, the full transcripts added up to 490 single-spaced pages. Second, a preliminary
coding grid covering the main themes of the study (the actual work of analysts, sources of
information, challenges of the rating process, ethical issues, relationships with companies,
controversies and scandals, and risk assessment) was developed based on the semi-
structured questionnaire. This grid was further improved by reading the transcripts and
holding discussions with other members of the research team on issues emerging from the
data. Each code was defined in order to improve the consistency in the coding process
between different coders (Thomas, 2006). Third, each transcript was categorized using the
coding grid. Two independent coders were involved in order to ensure the reliability of the
coding process (Miles et al., 2018; Thomas, 2006). Discussions between the coders allowed us
to further improve the grid by modifying, deleting, merging or splitting codes. The two
coders’ results were quite similar, and no significant differences were found. At the end of the
categorization process, 1,344 passages were coded under 58 different codes grouped into 11
main categories (see Appendix 2). To meet the objective of this study, those 11 categories were
further grouped into three main themes (challenges of sustainability ratings (SR),
characteristics of the analyst’s work and ethics). Fourth, the most relevant passages
related to the main categories were extracted. To comply with the research ethics protocol
and ensure the respondents’ confidentiality, all information related to the type of SRA or
organization involved was removed from the passages cited in this paper. Even though
quantification is not usually appropriate for qualitative methods (Gephart, 2004; Pratt, 2009),
certain trends based on the number of coded passages were estimated when deemed relevant.
Finally, the results of the research project were also presented to a group representing
industry members in a member verification effort to ensure the credibility and reliability of
the results (Birt et al., 2016).
The summary of the relevant findings and passages was organized into three main
themes, which structure the presentation of the findings. These three themes reflect the main
components of moral fictionalism described a posteriori in the literature review, namely (1) the
existence of collectively accepted moral or ethical judgments – in this case, regarding SRs as
tools for measuring and comparing sustainability performance; (2) the fictitious or
unsubstantiated nature of such judgments, which is highlighted by the analysis of the
major challenges of the SR process and the resulting trade-offs and (3) the practical utility of
these judgments, which is explored in the results analysis on the role of SRs in promoting
corporate sustainability.
AAAJ Opening the “black box” of sustainability rating agencies
34,8 Most respondents highlighted the growing trend in companies and investors taking
sustainable development or ESG issues into account, which has been accompanied by the
rapid development of SRA activities and the diversification of their services. Through their
rating activities, which are often carried out on behalf of large investors, SRAs are perceived
as essential actors in the development of SRIs and the promotion of CSR in general. These
trends were mentioned by about half of our respondents.
1750
For me, this is the first step in responsible investment. It is a tool to change ways of thinking, to raise
awareness of sustainability issues. This is a first step before disinvestment decisions.
(Respondent 27)
If you are a serious investor and you are going to invest a lot of money in something in the long term,
then you need to look at sustainability ratings, sustainability reports, and then you will ask
companies questions about their methods. (Respondent 24)
It’s expanding. We track about 7,000 public companies currently, and we’re expanding that out
pretty aggressively at the moment to probably about 13,000. (Respondent 2)
Unsurprisingly, the methods used to evaluate companies differ from one SRA to another and
are generally considered confidential; some respondents even called them a “trade secret” or
“recipe.” In this context, most analysts were reluctant to provide detailed information about
their methods and generally referred to documents freely available online, when such
documents existed. Despite the often laconic, imprecise and marketing-type nature of these
documents, they outline the services offered by the agencies and, in some cases (e.g. MSCI,
Vigeo Eiris), provide some vague information about elements of their methodology. Similarly,
the SR outcomes communicated to client companies do not contain clear descriptions of the
SRAs’ methodology. These SRs are mainly based on general data and comparisons (figures,
graphs, tables) on corporate sustainability in a specific sector, but they do not provide precise
or clear explanations of how the calculations were made.
Although the majority of respondents consider that differences in rating methods are
mainly responsible for the lack of convergence between SRAs, these methods are not
necessarily very precise, detailed or applicable in all circumstances. SRAs have considerably
expanded the number of activity sectors they cover, and now include increasingly diversified
sectors (education, health, agri-food, government agencies) that often require adjustments in
rating methods, sometimes on a case-by-case basis. As well, analysts are often asked to adapt
their evaluations to the particular requirements of their clients, who may request different
approaches or influence the weight given to specific sustainability criteria (e.g. pay equity,
climate change, occupational health and safety). These adaptations reflect different
conceptions of corporate sustainability and different priorities, depending on the
particularities of sectors or clients. The methods used are therefore not monolithic, static
or easily reproducible. They rather seem to be adaptable and customizable, and to evolve over
time depending on clients’ needs and broader institutional changes (e.g. new regulations, new
standards, increased pressure for certain issues). Monitoring these changes and adapting
rating practices to the evolving requirements of the institutional environment are two of the
many challenges of the multifaceted process of rating.
We’ve had a lot of new companies in the service sector, including educational companies, funeral
home companies, even some cultural event organization companies. New sectors entering the market
challenges us in terms of our methodology. We have to adjust to sectors to create new models for
them and quickly develop expertise on these sectors because we need to answer to investors’
requests as soon as we deliver these profiles. (Respondent 35)
 We will look at the indicators and evaluate whether or not our interpretation of the indicators has Black box of
changed, whether or not there are new laws that we must include in the indicators. (Respondent 16)
nonfinancial
It’s the investment fund in question that decides which issues are important to them. [. . .] If the agencies
environment is extremely important, I may give more weight to it. (Respondent 9)
In addition to possible adaptations in rating methods and criteria, the research and
interpretation activities involved in SR seem difficult to formalize. First, collecting
information on corporate sustainability is often a complex process that is based on 1751
nonstandardized data, and the exhaustiveness of the research process may vary. Some
respondents mentioned that the integrity and thoroughness of the information-gathering
process could vary considerably from one SRA to another, or even from one analyst to
another. In this context, the time and cost of quality research may conflict with the
economic objectives of SRAs, such as increasing the number of companies and sectors
covered by each analyst. The rapid growth of SRAs and difficulties finding qualified
analysts to meet the needs of an ever-increasing number of clients may therefore undermine
the quality and completeness of the analyses involved in the SR process. Second, despite the
rational and rigorous appearance of SRs, the work is partly based on personal
interpretations rather than a mechanical application of predefined “calculation recipes.”
About 69% of respondents acknowledged that SR analysts have to engage in some level of
judgment and interpretation at different stages of their work (e.g. data collection,
evaluation of the relevance and reliability of the information, interpretation of certain
events, controversy analysis). These interpretations inevitably involve ethical judgments
about where situations or activities should be placed on the range from acceptable to
reprehensible. Among other things, respondents highlighted the role of personal judgment,
the need to make trade-offs between several factors, a lesser formalization of sustainability
data compared to accounting information and the differences in interpretation between
several analysts.
It’s normal for a data provider to give more importance to one factor over another, due to their work
or their own experiences [. . .] Professionals are asked to make judgments. Some are probably better
than others, but that does not mean they’re necessarily wrong. (Respondent 19)
Even among us, there are differences of interpretation for difficult subjects. [. . .] And so, we have to
make a decision in this kind of debate and come to a consensus with the team, but not everyone
necessarily agrees. [. . .] When do we consider downgrading a company? This can also lead to
different interpretations, and it is not necessarily easy. (Respondent 36)

The bricolage of rating

The interpretative work and trade-offs underlying the activities of SR practitioners raise
questions about the meaning and reliability of the final rating given to each company. Several
analysts expressed regret that SR users limit themselves to the aggregated, simplified
sustainability score and do not seek access to the data and assessments supporting this
rating. Such an examination can indeed paint a very different picture of the company
depending what sustainability aspects were emphasized in calculating the aggregate score.
In addition to these aspects, the evaluation process raises many challenges, particularly in
terms of methodology, reliability of information, commercial issues and the availability of
relevant data.
First, about 47% of respondents mentioned the methodological challenges raised by the
rating process. These challenges include the complexity of the analyses, the particular
context of each organization and the difficulties in aggregating and comparing information
that is initially noncommensurable. Overall, the rating process requires in-depth knowledge
AAAJ of the companies, sectors and regions in order to be credible. Several respondents mentioned
34,8 the importance of integrating complex knowledge about the particularities of different
sectors of activity, the type of organization rated, the sustainability aspects analyzed or the
regions covered. Depending on the importance given to these various factors, the size of the
SRA or their activity portfolio, the analysts’ work can be divided into different dimensions
(e.g. specialization by region, sector, ESG aspects). However, integrating these factors and
aggregating the information obtained from very different contexts can be puzzling for SR
1752 practitioners. Among other things, respondents mentioned the importance of taking cultural,
regulatory and organizational differences into account in the analysis of data, particularly on
governance and social aspects. In addition, the data may be more or less aggregated, cover
very different activities (e.g. subsidiaries, divisions, geographical areas) and be focused on
indicators using different metrics. Sustainability information is often non-comparable,
contradictory, qualitative, context-dependent and focused on very different issues
(e.g. biodiversity, climate change, human rights). Processing this information to generate a
“clean” score that appears comparable resembles a sort of bricolage work based on many
approximations.
There are a lot of aspects that companies might seem to have in common that would easily give the
impression that they can be compared. However, sustainable development performance can be very
complex and, as many common elements as we can find between companies, our assessment can be
still challenged by very minor details that would somehow explain this impossibility. [. . .] The
equation for assessing sustainable development performance can be very complicated, no matter
how many variables we add to it to make it accurate. (Respondent 35)
If accountants are eventually further involved, it may allow for a more standardized methodology
and make it easier to compare companies. (Respondent 1)
Second, the reliability of the information used in the rating process remains questionable.
About 44% of respondents expressed doubts about the transparency of the data collected,
particularly due to greenwashing tendencies and companies concealing events or practices
that could negatively affect SRs. It can be time-consuming to verify the quality and reliability
of the information collected, especially when this requires the use of multiple sources and
contact with various stakeholders. Moreover, such verification is not always possible or
positively perceived by the companies being rated. Some SRAs are also reluctant to question
the quality of the data they obtain, including through their specific questionnaires.
Interestingly, some respondents mentioned – rightly or wrongly – that the assurance of
sustainability reports guarantees the reliability of the information obtained. Regardless of the
source of the information and how it is verified, the majority of respondents acknowledged
that SRAs are relatively vulnerable to the greenwashing and impression management
practices used by the companies being rated.
Look, if a company wants to hide something bad, they will unfortunately do it. So how can I detect it?
(Respondent 4)
If companies are not providing honest information, there’s very little a rating agency can do about it.
[. . .] There’s still the question of whether the information companies put out is accurate or not.
(Respondent 10)
I think that, today, there is still a lot of greenwashing and marketing behind the disclosure to make it
look good. [. . .] There are companies that do very well, but I think there are many others that just
conceal stuff. (Respondent 1)
Third, although most respondents highlighted the importance of the independence of SRAs
and of preventing conflicts of interest, about 33% mentioned pressures or situations likely to
compromise the impartiality of the rating process. Generally speaking, respondents were
open to describing their partnership with the rated companies and the importance of Black box of
maintaining good relations. Although this partnership seems necessary in order for the SRA nonfinancial
to obtain additional information on sustainability performance, it takes the form of exchanges
and meetings conducive to modifying SR outcomes or the SR criteria used. The majority of
agencies
respondents who mentioned such discussions or pressures to influence SR outcomes stressed
that it is essential, in order to maintain their credibility, that SRAs do not give in to these
pressures. Other respondents indicated that, in certain SRAs, ratings were validated by the
companies themselves or that commercial ties exist that could affect the impartiality of 1753
assessments. A few respondents also mentioned the consulting activities of SRAs, the
support they can provide to companies to help them improve their ratings and promotion of
the SRAs’ various services. It is worth noting that respondents did not explicitly acknowledge
that these activities may present further conflicts of interest.
We’re the only big agency that’s really independent. We’ve never been bought out, and we’ve never
bought other agencies. [. . .] We have no institution or company that could influence how we view the
results of our ratings. There are other players who are owned by the rated companies, so that’s
something that’s not acceptable to us. [. . .] We do not provide any advisory service to companies to
improve their ratings. Some agencies take a different view on this. (Respondent 12)
I guess when you audit and verify, you worry about potential conflict of interest, but how far they’ll
go in terms of saying it and then risk potentially not getting the contract renewed next year. . .
(Respondent 13)
There is a real mutual understanding, and people like us who rate companies contribute to the
dialogue. We also have group sessions [. . .] and we invite companies to come listen to us and present
our new products to them. (Respondent 30)
Fourth, about 22% of respondents mentioned difficulties in obtaining the information needed
to conduct SRs, irrespective of the reliability of the data collected. While the majority of
analysts acknowledged the usefulness of corporate sustainability reports, the information
these reports disclose is generally considered to be insufficient, difficult to use and not
necessarily focused on the most material issues. In addition, despite the increasing use of
reporting standards such as the GRI, the data disclosed often appear to be of little relevance,
poorly standardized or based on different metrics. It is therefore not necessarily tailored to the
SR process, which aims to make corporate sustainability performance clear, orderly and
comparable. To facilitate the collection and comparison of material information, most of
agencies studied send questionnaires to the rated companies. However, respondents
highlighted business fatigue around the proliferation of these questionnaires and the
difficulty of obtaining satisfactory responses. Responding to SRA questionnaires also
requires resources and skills that are not necessarily available in all companies. According to
respondents, vagueness and lack of information have a negative impact on the SR
assessment. As Respondent 24 summarized, “what has always bothered me in this industry is
that the quality of information, the quality of the message, is confused with the quality of
performance.” In this context, large companies with more resources are better positioned to
provide information to SRAs and are clearly favored over smaller ones. This also applies to
sectors or regions that are more familiar with SRA requirements.
Some sectors are much more difficult. There are also regions where there is no available information
that can be analyzed. This is one of the challenges we are working on all the time. (Respondent 16)
When a company does not disclose information, we traditionally penalize the company by
comparing the company with the worst case. [. . .] More than the importance of the numbers and the
information disclosed, I think it’s the absence of the information that makes the work really difficult,
especially for smaller companies. (Respondent 32)
AAAJ Promoting corporate sustainability?
34,8 Although the interviews conducted raise serious questions about the validity, meaning and
objectivity of SRs, the SRAs paradoxically seem to play an essential role in promoting
corporate sustainability. More specifically, despite their relative disconnection from the
complex, multifaceted and difficult-to-quantify realities of sustainability performance, SRs
appear to contribute to strengthening stakeholder inclusiveness, raising companies’ and
investors’ awareness of issues that are often neglected, and improving the quality of the
1754 information disclosed.
First, SRAs encourage the consideration of various stakeholders whose views are often
ignored in corporate sustainability reports. About 64% of respondents mentioned the
importance of using multiple sources of information and not limiting themselves to data
disclosed by companies. This approach is explicitly encouraged by the ARISTA 3.0
standard, which requires SRAs to “assess/rate companies using more than company-
provided information, such as independently collected data, regulatory sources, and
information from stakeholders or relevant NGOs” (Association for Responsible Investment
Service, 2012, p. 5). Although the process of consulting stakeholders and diversifying
information sources seems to vary greatly depending on the agencies and analysts
involved, it certainly helps to reduce the managerial capture of sustainability disclosure.
Several respondents also indicated that this approach helps to identify controversies that
may pose serious risks to investors. Nevertheless, several respondents emphasized that the
information shared by stakeholders is not necessarily more reliable than corporate
disclosures and that, when possible, it too needs to be verified. Moreover, it remains unclear
how conflicting and fuzzy information obtained from different sources is processed by the
rating system to produce a “clean” score.
It is very important to have other sources of information than just what comes from rated companies.
We take into account information provided by NGOs, the media, and our monitoring system on
controversies, scandals, etc. [. . .] We try to free ourselves as much as possible from the information
provided by rated companies. (Respondent 12)
It’s important for us to use a broad range of information sources. We know that’s just one
perspective, and that’s why we think it’s important to get a range from different stakeholders, such
as NGOs, or by looking at potential regulation breaches or press reports. (Respondent 15)
I believe in the benefits of taking stakeholders into account to promote shared decision-making and
to include various points of view. But how to apply this is still not necessarily clear to me.
(Respondent 4)
Second, about 31% of respondents mentioned that the SR process has a role in educating
managers and transforming corporate practices. SRA questionnaires and requests for
information help to raise managers’ awareness of indicators that were not measured or even
considered. Likewise, more informal meetings between SRA representatives and managers
may help companies learn how to better work with specific sustainability issues and
improve the company’s rating. Like financial results, poor SR results can stimulate
behavioral changes on the part of both rated companies and investors. Several respondents
pointed out that the rankings and evaluation reports that SRAs produce are clearer, more
concise and more effective tools for raising awareness than corporate sustainability
reports, which are often perceived as marketing tools. The figures and tables in SRs provide
a comforting impression of order and comparability that increases confidence in the
accuracy of measurements and makes SRs much easier to use than corporate sustainability
reports. Overall, the clarity of presentation of SRs and their illustrations (histograms,
examples, figures) contribute to helping their readers believe that the measurements are
objective, reliable and comparable. Moreover, SRs are only the most visible and best
advertised of the many activities of SRAs, which often offer other services to support Black box of
companies in working with sustainability issues (e.g. proxy voting, follow-up on improvements nonfinancial
and performance, support in engagement). In this context, some analysts view themselves
as agents of change or coaches engaged in a dialog or partnership with companies.
agencies
As a result, their mission goes far beyond simple performance evaluation, even if the latter
represents the bulk of their official work.
We might start a dialogue on a certain issue, let’s say labor rights, then they may end up wanting 1755
our input in general on their social responsibility agenda, how they should do their human rights
impact assessment, or something like that. They ask us for advice on how to do certain things.
[. . .] We tell the company that they need to improve their policy on human rights in order to get
certain key performance indicator fulfilled. Otherwise, they will not be fulfilled. And we explain
that to them. [. . .] That’s a way to engage and to make the company improve and change.
(Respondent 23)
I look for the best practices to have a pragmatic approach. Estimate the impact of a specific issue you
are worried about and go talk to the company. Show it to the director of investor relations, show him
the premises and your concerns, and ask how the company could address it. (Respondent 25)
Third, as mentioned by about 17% of respondents, SRAs contribute to improving companies’
sustainability disclosure practices. For instance, SRA requests for the information needed for
SRs may lead to the development of new indicators or the improvement of existing ones.
According to analysts, the inclusion of these indicators in sustainability reports or online
information tends to improve company performance, reduce the fatigue associated with SRA
questionnaires and facilitate the process of data collection. As well, some respondents said
that it is increasingly difficult for companies to hide important controversies or negative
events due to online information and the survey work conducted by SRAs. In this context,
SRAs tend to play a disciplinary role vis-a-vis some companies by encouraging the disclosure
of more transparent information on material issues.
I think things are changing in that companies are realizing that it’s very hard to hide information
these days because of technology and globalization. If you’ve got an operation in Asia, that’s not
to say somebody’s not going to find out if you’re behaving unethically there. You’ve got more
active civil society highlighting some of these issues, and it can impact what you do.
(Respondent 13)
Companies have changed their policies and procedures. Now they are more transparent, and they are
reporting more. (Respondent 25)
Public information has gaps. The level of ESG disclosure is generally quite limited. By talking with
companies, we get a little more information, and we can help to reduce these information gaps.
(Respondent 9)

Discussion
This paper sheds more light on the construction of SRs based on practitioners’ perceptions of
the validity, usefulness and ethical implications of the rating process using the theory of
moral fictionalism.
The results of the study show that, despite the rational appearance of SR practices, the
rating methods are relatively elastic and adaptable to sectoral or regional differences,
institutional changes and specific client demands. The diversification of the sectors covered
by SRAs and the complexity of sustainability issues do not favor a standardized or
monolithic approach. SRs thus depend, to a large extent, on the personal judgment and
professionalism of analysts rather than on an objective, reproducible, cast-in-stone approach.
The subjective and interpretative dimensions of the SR process are heightened by the
AAAJ difficulties in obtaining relevant information on the indicators measured, the uncertainties
34,8 surrounding the reliability of the information collected and the commensurability problems
that ensue from aggregating data from various sources on a wide array of issues. In addition,
the commercial aspects of SRAs and possible confusion between their assessment and
consulting functions invite questions about the independence and impartiality of SRs. In this
context, the objective and rational appearance of SRs can be considered to be a fiction – the
type of reassuring, accurate and accepted representation of reality pointed out by Vaihinger
1756 (2014) – but one that is necessary to ensure the legitimacy of SRs and respond to the demand
for information on the part of a growing number of rating users. Indeed, questioning the
reliability of SRs would represent a serious threat to the image of SRAs and the credibility of
SRIs in general. SRAs do not only act as translators by producing numerical information
from scattered and often fuzzy performance data (Avetisyan and Gond, 2013; Chelli and
Gendron, 2013), they also tend to create a fictional appearance of order, measurability and
comparability, which is necessary in order for SRIs to function properly and meet
stakeholders’ expectations. Independent of the reliability of SRs, SRAs play an important role
in institutionalizing sustainable development practices and standards. This
institutionalization is driven by companies and investors adapting to SR requirements and
criteria, as the latter exert a disciplinary power (Chelli and Gendron, 2013; Sauder and
Espeland, 2009). Certain SRAs, Vigeo Eiris in particular, also highlight their role in
integrating stakeholders’ perspectives, which are often neglected in sustainability
disclosures (Diouf and Boiral, 2017; Kolk, 2004; Perrini and Tencati, 2006).
This article contributes to the literature in several ways. First, the article contributes to the
emerging empirical literature on SRAs by opening the “black box” of rating practices by
exploring the perspectives of practitioners. With very few exceptions (D
ejean et al., 2004;
Sauder and Espeland, 2009), this view has been largely neglected in the empirical literature,
which is mainly based on secondary data and publicly available information on SR practices.
The results of this study thus contribute to demystifying the rating process and to developing
a better understanding of the many trade-offs underlying the fabric of SRs. Such trade-offs,
and the interpretative work involved in the rating process, can partly explain the opacity and
culture of secrecy surrounding SRA methodologies (Capelle-Blancard and Petit, 2017; Chelli
and Gendron, 2013; Delmas and Blass, 2010; Windolph, 2011). One can assume that these
methods could not be made public without questions being raised about their rigor and
objectivity, which would then be easy to verify and critique. The results of this study also
contribute to research on SRA convergence issues (Chatterji et al., 2016; Dorfleitner et al., 2015;
Semenova and Hassel, 2015). The methodological differences between SRAs are a partial and
convenient explanation for the lack of convergence this literature highlights. The divergence
between different SRs is also driven by the interpretative work involved in SRs, which is
shaped by the personal ethical judgments of SR analysts.
Second, this article contributes to an emerging body of literature that questions the
dominant practices in measuring corporate sustainability performance (Boiral and Henri,
2017; Capelle-Blancard and Petit, 2017; Delmas and Blass, 2010; Escrig-Olmedo et al., 2017;
Hubbard, 2009). Generally speaking, the measurement issues raised by respondents
(e.g. qualitative aspects, difficulties in obtaining reliable and material information, lack of
commensurable data, trade-offs in the weighting of different indicators or events) are similar to
those observed in the critical literature on sustainability performance (Boiral and Henri, 2017;
Gray and Milne, 2004; Talbot and Boiral, 2015). Our study shows that, whatever the methods
used, SRAs do not escape the challenges of the fact that sustainability is a multifaceted and
sometimes vague concept. This is obfuscated by the production of SR documents that contain
data, graphs and figures, the clear and easy-to-use appearance of which can obscure the trade-
offs underlying the measurement of corporate sustainability. These observations also
contribute to research on the use of figures and images as an impression management strategy
(Boiral, 2013; Cho et al., 2012a, b), which gives an appearance of comfort and order to the Black box of
complex, fuzzy and scattered nature of sustainability performance. nonfinancial
Third, this article contributes to the literature on SAR by showing the relevance of moral
fictionalism in analyzing the ambiguous role of SRs. To our knowledge, moral fictionalism
agencies
has not been explicitly used in the SAR field, although some critical studies on the subject are
based on analogous but not equivalent concepts, as this article details. Moral fictionalism is
employed here to shed light on the fictions and the philosophy of “as if” (Vaihinger, 2014)
underlying the rational appearance of ratings, while also illustrating ratings’ usefulness in 1757
promoting corporate sustainability. In general, the belief that ratings are rigorous and contain
comparable data on corporate sustainability is a response to broader institutional and
financial challenges, particularly the rapid growth of SRIs. These issues are very important,
and require that we seriously question the SRA outputs and the philosophy of “as if”
underlying their objective appearance. However, while the apparent rigor of SRs may appear
to be a fiction, SRs remain useful and even necessary for credibly promoting new standards
and practices among both organizations and investors. In this perspective, the study also
contributes to other fields, such as the research on organizational performance and the
disciplinary power of assessment practices (Chelli and Gendron, 2013; Christensen et al., 2013;
Sauder and Espeland, 2009; Siano et al., 2017).
Fourth, the results of the study have important practical implications for SRAs and SR
users. For SRAs, the difficulties in obtaining material and reliable information should
encourage competing agencies to collaborate in setting up common, more standardized
indicators that are easier for companies to work with. Likewise, pooling the questionnaires
sent to companies, in whole or in part, would help reduce companies’ fatigue and the costs
associated with information production. Although commercial differentiation strategies
may explain the current differences in SR methods, questionnaires and information required
from companies (Delmas and Blass, 2010; Saadaoui and Soobaroyen, 2018), such
differentiation could be focused on other aspects, such as emphasizing neglected
sustainability indicators, diversifying information sources, increasing stakeholder
involvement and preventing managerial capture (Diouf and Boiral, 2017; Kolk, 2004;
O’Dwyer and Owen, 2005; Perrini and Tencati, 2006). SRAs should also invest more in the
training and professionalization of their analysts. The interviews showed that some
analysts had a very approximate knowledge of major sustainability issues and were
unaware of the existence of disclosure standards such as the GRI. Respondents’ confidence
in the ability of assurance practices to improve information quality and to prevent
managerial capture also illustrates the need for better training in the field. It is worth
recalling that the effectiveness of assurance practices surrounding sustainability reports is
largely viewed as controversial in the literature (Boiral, 2013; O’Dwyer and Owen, 2005;
Smith et al., 2011; Talbot and Boiral, 2018). To improve their independence, SRAs need to
better separate their SR production activities from the services they offer to companies,
including consulting activities. In turn, SR users should more critically examine the
reliability of the ratings and challenge SRAs on their methodology and analyst training. The
use and comparison of several SR systems could potentially facilitate this step back and
reduce dependence on specific ranking methods that poorly represent the complexity
and diversity of sustainability issues. Given the increasing role of SRs in financial markets
and given that it is important for the appropriate information to be available for rating
purposes, companies should provide sufficient resources in order to better work with the
expectations of SRAs and adequately respond to their questionnaires. Lastly, investors,
public authorities and other stakeholders (e.g. NGOs and associations) should be cautious of
the real practical implications of SRs in light of the results of this study.
For future research, the influence of SRA methodology, size or geographical origin on SR
outputs could be analyzed by quantitative research using large samples. It would also be
AAAJ interesting to analyze the relationships between internal variables, such as the number of
34,8 companies covered by SRAs, the number of analysts involved and the type of SRs produced.
This type of study could also provide a better understanding of the factors that influence SR
quality. However, such studies are very difficult to conduct, mainly due to the “black box”
phenomenon that has been highlighted in the literature on SRAs (Delmas and Blass, 2010;
Rutledge, 2015; Saadaoui and Soobaroyen, 2018; Scalet and Kelly, 2010; Stubbs and Rogers,
2013; SustainAbility, 2018; Windolph, 2011). Second, while SRAs seem to play a useful role in
1758 developing SRI and promoting corporate sustainability, the implications of the uncertainties
and ambiguities surrounding SRs require further study. What do investors and SRI fund
managers think about the quality and reliability of the SRA outcomes? To what extent do
they care about the methods used by SRAs, their fictitious rationality and the lack of
convergence of their analyses? How do they justify such lack of convergence and what are its
possible implications for SRI practices? The relationship of companies with SRAs and the
influence of SRAs’ ratings on sustainability practices also merit further research. While SRAs
appear to exercise a disciplinary power over some companies, companies’ responses to SRA
requests for information may be mainly symbolic. The interviews revealed that the difficulty
in verifying the reliability of the information collected is one of the main challenges of the SR
process. In this context, it is reasonable to assume that some companies use impression
management strategies to influence SRAs, while others adopt a more transparent approach.
Although it is relevant to explore these issues, research on the subject remains difficult due to
the reluctance of SRAs, investors and companies to share information that may undermine
their legitimacy.

References
Adam, A.M. and Shavit, T. (2008), “How can a ratings-based method for assessing corporate social
responsibility (CSR) provide an incentive to firms excluded from socially responsible
investment indices to invest in CSR?”, Journal of Business Ethics, Vol. 82 No. 4, pp. 899-905.
Archel, P., Husillos, J. and Spence, C. (2011), “The institutionalisation of unaccountability: loading the
dice of Corporate Social Responsibility discourse”, Accounting, Organizations and Society,
Vol. 36 No. 6, pp. 327-343.
Association for Responsible Investment Service (2012), ARISTA 3.0, ARISE, Brussels.
Avetisyan, E. and Gond, J.-P. (2013), “Institutional dynamics of CSR standardization: a multilevel
perspective in the field of ESG rating”, Academy of Management Proceedings, Vol. 2013 No. 1,
p. 15122.
Avetisyan, E. and Hockerts, K. (2017), “The consolidation of the ESG rating industry as an enactment
of institutional retrogression”, Business Strategy and the Environment, Vol. 26 No. 3,
pp. 316-330.
Balaguer, M. (2009), “Fictionalism, theft, and the story of mathematics”, Philosophia Mathematica,
Vol. 17 No. 2, pp. 131-162.
Bebbington, J., Larrinaga, C. and Moneva, J.M. (2008), “Corporate social reporting and reputation risk
management”, Accounting, Auditing & Accountability Journal, Vol. 21 No. 3, pp. 337-361.
Benijts, T. (2008), “Measuring corporate sustainability: an analysis of the research practices used by
two Belgian asset management firms in the field of socially responsible investing”, Journal of
Corporate Citizenship No. 32, pp. 29-42.
Bessire, D. and Onn
ee, S. (2010), “Assessing corporate social performance: strategies of legitimation
and conflicting ideologies”, Critical Perspectives on Accounting, Vol. 21 No. 6, pp. 445-467.
Birt, L., Scott, S., Cavers, D., Campbell, C. and Walter, F. (2016), “Member checking: a tool to enhance
trustworthiness or merely a nod to validation?”, Qualitative Health Research, Vol. 26 No. 13,
pp. 1802-1811.
B€ohringer, C. and Jochem, P.E. (2007), “Measuring the immeasurable—a survey of sustainability Black box of
indices”, Ecological Economics, Vol. 63 No. 1, pp. 1-8.
nonfinancial
Boiral, O. (2013), “Sustainability reports as simulacra? A counter-account of A and Aþ GRI reports”,
Accounting, Auditing & Accountability Journal, Vol. 26 No. 7, pp. 1036-1071.
agencies
Boiral, O. (2016), “Accounting for the unaccountable: biodiversity reporting and impression
management”, Journal of Business Ethics, Vol. 135 No. 4, pp. 751-768.
Boiral, O. and Henri, J.F. (2017), “Is sustainability performance comparable? A study of GRI reports of 1759
mining organizations”, Business and Society, Vol. 56 No. 2, pp. 283-317.
Bowen, F. (2014), After Greenwashing: Symbolic Corporate Environmentalism and Society, Cambridge
University Press, Cambridge.
Bowen, F. and Aragon-Correa, J.A. (2014), “Greenwashing in corporate environmentalism research and
practice: the importance of what we say and do”, Organization and Environment, Vol. 27 No. 2,
pp. 107-112.
Brown, H.S., de Jong, M. and Levy, D.L. (2009), “Building institutions based on information disclosure:
lessons from GRI’s sustainability reporting”, Journal of Cleaner Production, Vol. 17 No. 6,
pp. 571-580.
Brunsson, N. (1989), The Organization of Hypocrite: Talk, Decisions and Actions in Organizations, John
Wiley and Sons, Chichester.
Capelle-Blancard, G. and Petit, A. (2017), “The weighting of CSR dimensions: one size does not fit all”,
Business and Society, Vol. 56 No. 6, pp. 919-943.
Capelle-Blancard, G. and Monjon, S. (2014), “The performance of socially responsible funds: does the
screening process matter?”, European Financial Management, Vol. 20 No. 3, pp. 494-520.
Chatterji, A. and Levine, D. (2006), “Breaking down the wall of codes: evaluating non-financial
performance measurement”, California Management Review, Vol. 48 No. 2, pp. 29-51.
Chatterji, A.K., Levine, D.I. and Toffel, M.W. (2009), “How well do social ratings actually measure
corporate social responsibility?”, Journal of Economics and Management Strategy, Vol. 18 No. 1,
pp. 125-169.
Chatterji, A.K., Durand, R., Levine, D.I. and Touboul, S. (2016), “Do ratings of firms converge?
Implications for managers, investors and strategy researchers”, Strategic Management Journal,
Vol. 37 No. 8, pp. 1597-1614.
Chelli, M. and Gendron, Y. (2013), “Sustainability ratings and the disciplinary power of the ideology of
numbers”, Journal of Business Ethics, Vol. 112 No. 2, pp. 187-203.
Cho, C.H. and Patten, D.M. (2007), “The role of environmental disclosures as tools of legitimacy: a
research note”, Accounting, Organizations and Society, Vol. 32 No. 7, pp. 639-647.
Cho, C.H., Michelon, G. and Patten, D.M. (2012a), “Enhancement and obfuscation through the use of
graphs in sustainability reports: an international comparison”, Sustainability Accounting,
Management and Policy Journal, Vol. 3 No. 1, pp. 74-88.
Cho, C.H., Michelon, G. and Patten, D.M. (2012b), “Impression management in sustainability reports:
an empirical investigation of the use of graphs”, Accounting and the Public Interest, Vol. 12
No. 1, pp. 16-37.
Cho, C.H., Laine, M., Roberts, R.W. and Rodrigue, M. (2015), “Organized hypocrite, organizational
façades, and sustainability reporting”, Accounting, Organizations and Society, Vol. 40, pp. 78-94.
Christensen, L.T., Morsing, M. and Thyssen, O. (2013), “CSR as aspirational talk”, Organization,
Vol. 20 No. 3, pp. 372-393.
Corbin, J.M. and Strauss, A. (1990), “Grounded theory research: procedures, canons, and evaluative
criteria”, Qualitative Sociology, Vol. 13 No. 1, pp. 3-21.
AAAJ Crilly, D., Hansen, M. and Zollo, M. (2016), “The grammar of decoupling: a cognitive-linguistic
perspective on firms’ sustainability claims and stakeholders’ interpretation”, Academy of
34,8 Management Journal, Vol. 59 No. 2, pp. 705-729.
D
ejean, F., Gond, J.P. and Leca, B. (2004), “Measuring the unmeasured: an institutional entrepreneur
strategy in an emerging industry”, Human Relations, Vol. 57 No. 6, pp. 741-764.
D
ejean, F., Giamporcaro, S., Gond, J.-P., Leca, B. and Penalva-Icher, E. (2013), “Mistaking an emerging
market for a social movement? A comment on Arjalies’ social-movement perspective on socially
1760 responsible investment in France”, Journal of Business Ethics, Vol. 112 No. 2, pp. 205-212.
De Villiers, C. and Van Staden, C. (2011), “Where firms choose to disclose voluntary environmental
information”, Journal of Accounting and Public Policy, Vol. 30 No. 6, pp. 504-525.
Deegan, C. (2010), “Organizational legitimacy as a motive for sustainability reporting”, in Unerman, J.,
Bebbington, J. and O’Dwyer, B. (Eds), Sustainability Accounting and Accountability, Routledge,
New York, NY, pp. 146-168.
Delmas, M. and Blass, V.D. (2010), “Measuring corporate environmental performance: the trade-offs of
sustainability ratings”, Business Strategy and the Environment, Vol. 19 No. 4, pp. 245-260.
Devinney, T.M. (2009), “Is the socially responsible corporation a myth? The good, the bad, and the
ugly of corporate social responsibility”, Academy of Management Perspectives, Vol. 23 No. 2,
pp. 44-56.
Dillenburg, S., Greene, T. and Erekson, O.H. (2003), “Approaching socially responsible investment
with a comprehensive ratings scheme: total social impact”, Journal of Business Ethics, Vol. 43
No. 3, pp. 167-177.
Diouf, D. and Boiral, O. (2017), “The quality of sustainability reports and impression management: a
stakeholder perspective”, Accounting, Auditing & Accountability Journal, Vol. 30 No. 3,
pp. 643-667.
Dorfleitner, G., Halbritter, G. and Nguyen, M. (2015), “Measuring the level and risk of corporate
responsibility–An empirical comparison of different ESG rating approaches”, Journal of Asset
Management, Vol. 16 No. 7, pp. 450-466.
Dunne, C. (2011), “The place of the literature review in grounded theory research”, International
Journal of Social Research Methodology, Vol. 14 No. 2, pp. 111-124.
Eccles, R.G. and Stroehle, J.C. (2018), “Exploring social origins in the construction of ESG measures”,
Social Science Research Network. doi: 10.2139/ssrn.3212685.
Eklund, M. (2017), “Fictionalism”, in Zalta, E.N. (Ed.), Stanford Encyclopedia of Philosophy, available at:
https://plato.stanford.edu/archives/win2017/entries/fictionalism/.
Elbasha, T. and Avetisyan, E. (2018), “A framework to study strategizing activities at the field level:
the example of CSR rating agencies”, European Management Journal, Vol. 36 No. 1, pp. 38-46.
Entine, J. (2003), “The myth of social investing: a critique of its practice and consequences for
corporate social performance research”, Organization and Environment, Vol. 16 No. 3,
pp. 352-368.
Escrig-Olmedo, E., Mu~ noz-Torres, M.J. and Fernandez-Izquierdo, M.A. (2010), “Socially responsible
investing: sustainability indices, ESG rating and information provider agencies”, International
Journal of Sustainable Economy, Vol. 2 No. 4, pp. 442-461.
Escrig-Olmedo, E., Mu~
and Rivera-Lirio, J.M. (2014),
noz-Torres, M.J., Fern
andez-Izquierdo, M.A.
“Lights and shadows on sustainability rating scoring”, Review of Managerial Science, Vol. 8
No. 4, pp. 559-574.
Escrig-Olmedo, E., Mu~
and Rivera-Lirio, J.M. (2017),
noz-Torres, M.J., Fern
andez-Izquierdo, M.A.
“Measuring corporate environmental performance: a methodology for sustainable
development”, Business Strategy and the Environment, Vol. 26 No. 2, pp. 142-162.
Fine, A. (1993), “Fictionalism”, Midwest Studies in Philosophy, Vol. 18 No. 1, pp. 1-18.
Fowler, S.J. and Hope, C. (2007), “A critical review of sustainable business indices and their impact”, Black box of
Journal of Business Ethics, Vol. 76 No. 3, pp. 243-252.
nonfinancial
Friedman, A.L. and Miles, S. (2001), “Socially responsible investment and corporate social and
environmental reporting in the UK: an exploratory study”, The British Accounting Review,
agencies
Vol. 33 No. 4, pp. 523-548.
Gauthier, J. and Wooldridge, B. (2018), “Sustainability ratings and organizational legitimacy: the role
of compensating tactics”, in Gal, G., Akisik, O. and Wooldridge, W. (Eds), Sustainability and
Social Responsibility: Regulation and Reporting, Springer, Singapore, pp. 141-157. 1761
Gendron, Y. (2018), “On the elusive nature of critical (accounting) research”, Critical Perspectives on
Accounting, Vol. 50, pp. 1-12.
Gephart, R.P. (2004), “Qualitative research and the academy of management journal”, Academy of
Management Journal, Vol. 47 No. 4, pp. 454-462.
Gibassier, D., Rodrigue, M. and Arjalies, D.-L. (2018), “Integrated reporting is like God: no one has met
Him, but everybody talks about Him”, Accounting, Auditing & Accountability Journal, Vol. 31
No. 5, pp. 1349-1380.
Global Sustainable Investment Alliance (2016), “2016 global sustainable investment review”, GSIA,
available at: http://www.gsi-alliance.org/wp-content/uploads/2017/03/GSIR_Review2016.F.pdf.
Graafland, J.J., Eijffinger, S.C. and SmidJohan, H. (2004), “Benchmarking of corporate social
responsibility: methodological problems and robustness”, Journal of Business Ethics, Vol. 53
Nos 1-2, pp. 137-152.
Grandy, G. and Mills, A.J. (2004), “Strategy as simulacra? A radical reflexive look at the discipline and
practice of strategy”, Journal of Management Studies, Vol. 41 No. 7, pp. 1153-1170.
Gray, R. and Milne, M. (2004), “Towards reporting on the triple bottom line: mirages, methods and
myths”, in Henriques, A. and Richardson, J. (Eds), The Triple Bottom Line: Does it All Add up,
Earthscan, London, pp. 70-80.
Guthrie, J. and Parker, L.D. (1989), “Corporate social reporting: a rebuttal of legitimacy theory”,
Accounting and Business Research, Vol. 19 No. 76, pp. 343-352.
Hahn, R. and K€uhnen, M. (2013), “Determinants of sustainability reporting: a review of results, trends,
theory, and opportunities in an expanding field of research”, Journal of Cleaner Production,
Vol. 59, pp. 5-21.
Hahn, R. and L€ulfs, R. (2014), “Legitimizing negative aspects in GRI-oriented sustainability reporting:
a qualitative analysis of corporate disclosure strategies”, Journal of Business Ethics, Vol. 123
No. 3, pp. 401-420.
Hambrick, D.C. (2007), “The field of management’s devotion to theory: too much of a good thing?”,
Academy of Management Journal, Vol. 50 No. 6, pp. 1346-1352.
Haniffa, R.M. and Cooke, T.E. (2005), “The impact of culture and governance on corporate social
reporting”, Journal of Accounting and Public Policy, Vol. 24 No. 5, pp. 391-430.
Hedesstr€om, M., Lundqvist, U. and Biel, A. (2011), “Investigating consistency of judgement across
sustainability analyst organizations”, Sustainable Development, Vol. 19 No. 2, pp. 119-134.
Heggen, C., Sridharan, V. and Subramaniam, N. (2018), “To the letter vs the spirit: a case analysis of
contrasting environmental management responses”, Accounting, Auditing & Accountability
Journal, Vol. 31 No. 2, pp. 478-502.
Higgins, C. and Larrinaga, C. (2014), “Sustainability reporting: insights from institutional theory”, in
Bebbington, J., Unerman, J. and O’Dwyer, B. (Eds), Sustainability Accounting and Accountability,
Routledge, New York, NY, pp. 291-303.
Holt, A. (2010), “Using the telephone for narrative interviewing: a research note”, Qualitative Research,
Vol. 10 No. 1, pp. 113-121.
Hubbard, G. (2009), “Measuring organizational performance: beyond the triple bottom line”, Business
Strategy and the Environment, Vol. 18 No. 3, pp. 177-191.
AAAJ Igalens, J. and Gond, J.P. (2005), “Measuring corporate social performance in France: a critical and
empirical analysis of ARESE data”, Journal of Business Ethics, Vol. 56 No. 2, pp. 131-148.
34,8
Joyce, R. (2001), The Myth of Morality, Cambridge University Press, Cambridge.
Joyce, R. (2005), “Moral fictionalism”, in Kalderon, M.E. (Ed.), Fictionalism in Metaphysics, Clarendon
Press, Broadbridge, pp. 287-313.
Kalderon, M.E. (2005), Moral Fictionalism, Oxford University Press, New York, NY.
1762 Kim, S. (2006), “Moral realism”, The Internet Encyclopedia of Philosophy, available at: https://www.iep.
utm.edu/moralrea/.
Koellner, T., Weber, O., Fenchel, M. and Scholz, R. (2005), “Principles for sustainability rating of
investment funds”, Business Strategy and the Environment, Vol. 14 No. 1, pp. 54-70.
Kolk, A. (2004), “A decade of sustainability reporting: developments and significance”, International
Journal of Environmental and Sustainable Development, Vol. 3 No. 1, pp. 51-64.
Kotsantonis, S., Pinney, C. and Serafeim, G. (2016), “ESG integration in investment management:
myths and realities”, Journal of Applied Corporate Finance, Vol. 28 No. 2, pp. 10-16.
Lackmann, J., Ernstberger, J. and Stich, M. (2012), “Market reactions to increased reliability of
sustainability information”, Journal of Business Ethics, Vol. 107 No. 2, pp. 111-128.
Larrinaga Gonz
alez, C. (2017), “En torno a la evoluci
on de la investigaci
on en contabilidad social y
medioambiental”, Revista Latinoamericana de Investigaci
on en Organizaciones, Ambiente y
Sociedad, Vol. 8 No. 11, pp. 21-38.
Liern, V. and P
erez-Gladish, B. (2018), “Companies’ selection methods for inclusion in sustainable
indices: a fuzzy approach”, in Berger-Vachon, C., Gil Lafuente, A.M., Kacprzyk, J., Kondratenko,
Y., Merig
o, J.M. and Morabito, C.F. (Eds), Complex Systems: Solutions and Challenges in
Economics, Management and Engineering, Springer, Cham, pp. 365-380.
Liu, K.F.R. (2007), “Evaluating environmental sustainability: an integration of multiple-criteria
decision-making and fuzzy logic”, Environmental Management, Vol. 39 No. 5, pp. 721-736.
Martin, P.Y. and Turner, B.A. (1986), “Grounded theory and organizational research”, The Journal of
Applied Behavioral Science, Vol. 22 No. 2, pp. 141-157.
Mattingly, J.E. and Berman, S.L. (2006), “Measurement of corporate social action: discovering
taxonomy in the Kinder Lydenburg Domini ratings data”, Business and Society, Vol. 45 No. 1,
pp. 20-46.
Merkl-Davies, D.M. and Brennan, N.M. (2007), “Discretionary disclosure strategies in corporate
narratives: incremental information or impression management?”, Journal of Accounting
Literature, Vol. 27, pp. 116-196.
Merkl-Davies, D.M., Brennan, N.M. and McLeay, S.J. (2011), “Impression management and
retrospective sense-making in corporate narratives: a social psychology perspective”,
Accounting, Auditing & Accountability Journal, Vol. 24 No. 3, pp. 315-344.
Meyer, J.W. and Rowan, B. (1977), “Institutionalized organizations: formal structure as myth and
ceremony”, American Journal of Sociology, Vol. 83 No. 2, pp. 340-363.
Miles, M.B., Huberman, A.M. and Saldana, J. (2018), Qualitative Data Analysis: A Methods Sourcebook,
Sage Publications, Thousand Oaks, CA.
Milne, M.J., Kearins, K. and Walton, S. (2006), “Creating adventures in wonderland: the journey
metaphor and environmental sustainability”, Organization, Vol. 13 No. 6, pp. 801-839.
Moneva, J.M., Archel, P. and Correa, C. (2006), “GRI and the camouflaging of corporate
unsustainability”, Accounting Forum, Vol. 30 No. 2, pp. 121-137.
Mooij, S. (2017), “The ESG rating and ranking industry; Vice or virtue in the adoption of responsible
investment?”, Social Science Research Network. doi: 10.2139/ssrn.2960869.
Mu~
Rivera-Lirio, J.M. and Escrig-Olmedo, E. (2019), “Can
noz-Torres, M.J., Fern
andez-Izquierdo, M.A.,
environmental, social, and governance rating agencies favor business models that promote a
 more sustainable development?”, Corporate Social Responsibility and Environmental Black box of
Management, Vol. 26 No. 2, pp. 439-452.
nonfinancial
Nolan, D., Restall, G. and West, C. (2005), “Moral fictionalism versus the rest”, Australasian Journal of
Philosophy, Vol. 83 No. 3, pp. 307-330.
agencies
Novethic Research (2014), Overview of ESG Rating Agencies, Novethic, Paris, France, available at:
https://www.novethic.com/fileadmin/user_upload/tx_ausynovethicetudes/pdf_complets/2014_
Overview-of-ESG-rating-agencies.pdf.
1763
O’Dwyer, B. and Owen, D.L. (2005), “Assurance statement practice in environmental, social and
sustainability reporting: a critical evaluation”, The British Accounting Review, Vol. 37 No. 2,
pp. 205-229.
Owen, D. (2006), “Emerging issues in sustainability reporting”, Business Strategy and the
Environment, Vol. 15 No. 4, pp. 217-218.
Parguel, B., Beno^ıt-Moreau, F. and Larceneux, F. (2011), “How sustainability ratings might deter
‘greenwashing’: a closer look at ethical corporate communication”, Journal of Business Ethics,
Vol. 102 No. 1, pp. 15-28.
P
erez-L
opez, D., Moreno-Romero, A. and Barkemeyer, R. (2015), “Exploring the relationship between
sustainability reporting and sustainability management practices”, Business Strategy and the
Environment, Vol. 24 No. 8, pp. 720-734.
Perkiss, S., Bernardi, C., Dumay, J. and Haslam, J. (2020), “A sticky chocolate problem: impression
management and counter accounts in the shaping of corporate image”, Critical Perspectives on
Accounting, p. 102229.
Perrini, F. and Tencati, A. (2006), “Sustainability and stakeholder management: the need for new
corporate performance evaluation and reporting systems”, Business Strategy and the
Environment, Vol. 15 No. 5, pp. 296-308.
Pfeffer, J. (2007), “A modest proposal: how we might change the process and product of managerial
research”, Academy of Management Journal, Vol. 50 No. 6, pp. 1334-1345.
Phillips, N. (1995), “Telling organizational tales: on the role of narrative fiction in the study of
organizations”, Organization Studies, Vol. 16 No. 4, pp. 625-649.
Pratt, M.G. (2009), “From the editors: for the lack of a boilerplate: tips on writing up (and reviewing)
qualitative research”, Academy of Management Journal, Vol. 52 No. 5, pp. 856-862.
Rahaman, A.S., Lawrence, S. and Roper, J. (2004), “Social and environmental reporting at the VRA:
institutionalised legitimacy or legitimation crisis?”, Critical Perspectives on Accounting, Vol. 15
No. 1, pp. 35-56.
Rahdari, A.H. and Rostamy, A.A.A. (2015), “Designing a general set of sustainability indicators at the
corporate level”, Journal of Cleaner Production, Vol. 108, pp. 757-771.
Rajak, S. and Vinodh, S. (2015), “Application of fuzzy logic for social sustainability performance
evaluation: a case study of an Indian automotive component manufacturing organization”,
Journal of Cleaner Production, Vol. 108, pp. 1184-1192.
Rasche, A. and Behnam, M. (2009), “As if it were relevant: a systems theoretical perspective on the
relation between science and practice”, Journal of Management Inquiry, Vol. 18 No. 3,
pp. 243-255.
Richardson, B.J. (2009), “Keeping ethical investment ethical: regulatory issues for investing for
sustainability”, Journal of Business Ethics, Vol. 87 No. 4, pp. 555-572.
Roberts, J. (2018), “Managing only with transparency: the strategic functions of ignorance”, Critical
Perspectives on Accounting, Vol. 55, pp. 53-60.
Rutledge, E.L. (2015), How Do CSR Rating Schemes Influence Corporate Behavior?: Lessons from the
Utility Industry, Massachusetts Institute of Technology, Cambridge, MA.
Saadaoui, K. and Soobaroyen, T. (2018), “An analysis of the methodologies adopted by CSR rating
agencies”, Sustainability Accounting, Management and Policy Journal, Vol. 9 No. 1, pp. 43-62.
AAAJ Santilli, P. (1984), “Moral fictions and scientific management”, Journal of Business Ethics, Vol. 3 No. 4,
pp. 279-285.
34,8
Sauder, M. and Espeland, W.N. (2009), “The discipline of rankings: tight coupling and organizational
change”, American Sociological Review, Vol. 74 No. 1, pp. 63-82.
Savage, P., Cornelissen, J.P. and Franck, H. (2018), “Fiction and organization studies”, Organization
Studies, Vol. 39 No. 7, pp. 975-994.
1764 Scalet, S. and Kelly, T.F. (2010), “CSR rating agencies: what is their global impact?”, Journal of
Business Ethics, Vol. 94 No. 1, pp. 69-88.
Sch€afer, H. (2005), “International corporate social responsibility rating systems: conceptual outline and
empirical results”, Journal of Corporate Citizenship No. 20, pp. 107-120.
Searcy, C. and Elkhawas, D. (2012), “Corporate sustainability ratings: an investigation into how
corporations use the Dow Jones sustainability index”, Journal of Cleaner Production, Vol. 35,
pp. 79-92.
Semenova, N. and Hassel, L.G. (2015), “On the validity of environmental performance metrics”, Journal
of Business Ethics, Vol. 132 No. 2, pp. 249-258.
Siano, A., Vollero, A., Conte, F. and Amabile, S. (2017), “‘More than words’: expanding the taxonomy of
greenwashing after the Volkswagen scandal”, Journal of Business Research, Vol. 71, pp. 27-37.
Singh, R.K., Murty, H.R., Gupta, S.K. and Dikshit, A.K. (2012), “An overview of sustainability
assessment methodologies”, Ecological Indicators, Vol. 15 No. 1, pp. 281-299.
Smith, J., Haniffa, R. and Fairbrass, J. (2011), “A conceptual framework for investigating ‘capture’ in
corporate sustainability reporting assurance”, Journal of Business Ethics, Vol. 99 No. 3,
pp. 425-439.
Sodjahin, A., Champagne, C., Coggins, F. and Gillet, R. (2017), “Leading or lagging indicators of risk?
The informational content of extra-financial performance scores”, Journal of Asset
Management, Vol. 18 No. 5, pp. 347-370.
Solomon, J.F., Solomon, A., Joseph, N.L. and Norton, S.D. (2013), “Impression management, myth
creation and fabrication in private social and environmental reporting: insights from Erving
Goffman”, Accounting, Organizations and Society, Vol. 38 No. 3, pp. 195-213.
Stephens, N. (2007), “Collecting data from elites and ultra elites: telephone and face-to-face interviews
with macroeconomists”, Qualitative Research, Vol. 7 No. 2, pp. 203-216.
Strauss, A. and Corbin, J. (1994), “Grounded theory methodology”, Handbook of Qualitative Research,
Vol. 17, pp. 273-285.
Stubbs, W. and Rogers, P. (2013), “Lifting the veil on environment-social-governance rating methods”,
Social Responsibility Journal, Vol. 9 No. 4, pp. 622-640.
Sturges, J.E. and Hanrahan, K.J. (2004), “Comparing telephone and face-to-face qualitative
interviewing: a research note”, Qualitative Research, Vol. 4 No. 1, pp. 107-118.
Suddaby, R. (2006), “From the editors: what grounded theory is not”, Academy of Management
Journal, Vol. 49 No. 4, pp. 633-642.
SustainAbility (2018), “Rate the raters 2018: ratings revisited”, SustainAbility, London, New York,
San Francisco, available at: https://sustainability.com/wp-content/uploads/2017/12/SA-
RateTheRaters_Ratings-Revisited_March18.pdf.
Talbot, D. and Boiral, O. (2015), “Strategies for climate change and impression management: a case
study among Canada’s large industrial emitters”, Journal of Business Ethics, Vol. 132 No. 2,
pp. 329-346.
Talbot, D. and Boiral, O. (2018), “GHG reporting and impression management: an assessment of
sustainability reports from the energy sector”, Journal of Business Ethics, Vol. 147 No. 2,
pp. 367-383.
Thomas, D.R. (2006), “A general inductive approach for analyzing qualitative evaluation data”, Black box of
American Journal of Evaluation, Vol. 27 No. 2, pp. 237-246.
nonfinancial
Thomas, A. (2012), “Nietzsche and moral fictionalism”, in Janaway, C. and Robertson, S. (Eds),
Nietzsche, Naturalism, and Normativity, Oxford University Press, Oxford, pp. 133-159.
agencies
Vaihinger, H. (2014), The Philosophy of “As if”, Routledge, Oxon.
Van Den Brink, T.W. and van Der Woerd, F. (2004), “Industry specific sustainability benchmarks: an
ECSF pilot bridging corporate sustainability with social responsible investments”, Journal of 1765
Business Ethics, Vol. 55 No. 2, pp. 187-203.
Velez-Castrillon, S. (2014), “Socially responsible investment rating standards: who rates the raters?”,
Journal of International Business Disciplines, Vol. 9 No. 2, pp. 1-18.
Windolph, S.E. (2011), “Assessing corporate sustainability through ratings: challenges and their
causes”, Journal of Environmental Sustainability, Vol. 1 No. 1, p. 5.
Zbaracki, M.J. (1998), “The rhetoric and reality of total quality management”, Administrative Science
Quarterly, Vol. 43 No. 3, pp. 602-636.
AAAJ Appendix 1
34,8

Respondent Title Specialty Organization type Country

1 CEO and Accounting SME Canada

lecturer
1766 2 Director Sustainability data Database provider United States
3 Analyst Nonfinancial rating Service provider Germany
4 Professional Responsible investment and proxy voting Service provider Canada
5 Analyst Sustainability data Database provider Germany
6 Director Responsible investment Institutional United
investor Kingdom
7 Analyst Nonfinancial rating Service provider Germany
8 Analyst Responsible investment Institutional France
investor
9 Professional Responsible investment and proxy voting Institutional Canada
investor
10 Manager Nonfinancial rating Service provider United
Kingdom
11 Vice-president Nonfinancial rating Service provider Canada
12 Vice-president Nonfinancial rating Service provider France
13 Professional Responsible investment Institutional United
investor Kingdom
14 Voting agent Responsible investment and proxy voting Institutional France
investor
15 Analyst Responsible investment Institutional United
investor Kingdom
16 Manager Nonfinancial rating Service provider Canada
17 Voting agent Responsible investment and proxy voting Service provider Canada
18 Analyst Responsible investment Institutional Canada
investor
19 Professional Responsible investment and shareholder Credit union Canada
engagement
20 Professional Responsible investment and sustainability Credit union Canada
integration
21 Analyst Sustainability integration Nonprofit Canada
organization
22 Lecturer Responsible investment Academic Canada
23 Manager Shareholder engagement Service provider Sweden
24 Director Shareholder engagement Service provider Canada
25 Analyst Responsible investment Institutional Brazil
investor
26 Director Shareholder engagement Service provider Canada
27 Voting agent Responsible investment and proxy voting Service provider Canada
28 Consultant Responsible investment and nonfinancial Service provider Canada
rating
29 Manager Nonfinancial rating Service provider United States
30 CEO Shareholder engagement and proxy voting Service provider France
31 Analyst Nonfinancial rating Service provider India
32 Analyst Nonfinancial rating Nonprofit Brazil
organization
33 Manager Sustainability data Database provider United
Kingdom
34 Vice-president Nonfinancial rating Service provider Sweden
Table A1. 35 Analyst Nonfinancial rating Service provider Morocco
Profile of the 36 Manager Responsible investment and proxy voting Institutional France
interviewees investor
Appendix 2 Black box of
nonfinancial
agencies
Categories (n 5 11) Codes (n 5 58)

Activism and drivers (1) Decarbonization

(2) Dialog 1767
(3) Divestment
(4) Impact of activism and drivers
(5) Internal drivers
(6) Involvement in annual general meetings
(7) Practical examples
(8) Proxy voting
(9) Shareholder engagement
(10) Shareholder proposal
Challenges of sustainability ratings (SRs) (1) Aggregation into a single indicator
(2) Challenges of voluntary disclosure
(3) Credibility and reliability of SRs
(4) Credibility of information contained in sustainability
reports
(5) Difficulty in comparing information
(6) Greenwashing and storytelling
(7) Lack of balance
(8) Mechanisms of verification
(9) Nonfungibility of performance
(10) Other challenges
(11) Reasons for the lack of convergence between agencies
(12) Standardization of data
(13) Standardization of norm
(14) SRs from other companies
Characteristics of the analysts’ work (1) Controversy assessment
(2) Data collection
(3) Evaluation, use and updating of information
(4) Responsibilities
(5) Typical day/task
(6) Writing
Conflicts of interest in SRs (1) Solicited ratings
(2) Unsolicited ratings
Controversies and scandals (1) Consequences
(2) Examples
(3) Impact on rating
(4) Prediction of scandals
Discussion with rated companies (1) Additional information sent
(2) Impact of discussion on the ratings
(3) Validation of the ratings
Ethics (1) Accountability
(2) Conflicts of interest
(3) Transparency
Negative aspects of the contact with rated (1) Information request saturation
companies (2) Pressures and ethical issues
Responses from rated companies (1) Collaboration and openness Table A2.
(2) Providing of explanations Categories and codes
identified in the
(continued ) categorization process
AAAJ Categories (n 5 11) Codes (n 5 58)
34,8
Risks (1) Environmental risks
(2) Impacts on ratings
(3) Legal risks
(4) Practical examples
(5) Reliability of risk assessment
1768 (6) Risk assessment process
Sources of information (1) Company’s information
(2) Counter-accounting process
(3) Government information
(4) Legal proceedings
(5) Other sources
Table A2. (6) Proprietary system

About the authors

Olivier Boiral is a Professor at the Faculty of Business Administration of Universit
e Laval (Qu
ebec City)
and holds the Canada Research Chair in the Internalization of Sustainability Practices and
Organizational Accountability. Recipient of a PhD from HEC Montreal, his recent research has
focused on the critical analysis of the sustainable development concept, sustainability reports concept
and the implementation of international management standards. His most important papers have been
published in Organization Studies, Organization Science, Long Range Planning, International Journal of
Management Reviews, Business and Society and Journal of Business Ethics.


David Talbot is a Professor at the Ecole nationale d’administration publique (ENAP) in Quebec City.
Recipient of a PhD from Universit
e Laval, his current research mainly focuses on the analysis of
environmental policies and environmental disclosure in the public sector. His most important papers
have been published in Journal of Business Ethics, Energy Policy and Business Strategy and the
Environment.
Marie-Christine Brotherton is a Research Assistant at the Faculty of Business Administration of
Universit
e Laval (Qu
ebec City). She holds a PhD in Cellular and Molecular Biology and an MBA in
Corporate Social Responsibility from Universit
e Laval. Her recent research focuses on several CSR
practices, such as the management of biodiversity issues and relationships with indigenous peoples,
voluntary certifiable standards and the assurance of sustainability reports.
I~naki Heras-Saizarbitoria (PhD, University of the Basque Country, Bilbao) is a Professor of the
Department of Business Management at the University of the Basque Country in San Sebastian (Spain).
His recent research has focused on the critical analysis of the adoption of management practices, such as
Voluntary certifiable standards, Environmental Management practices and CSR practices. He has
published several articles in international academic journals (i.e. Organization, International Journal
of Management Reviews, Business Strategy and the Environment and Journal of Business Ethics).
I~
naki Heras-Saizarbitoria is the corresponding author and can be contacted at: [email protected]

For instructions on how to order reprints of this article, please visit our website:
www.emeraldgrouppublishing.com/licensing/reprints.htm
Or contact us for further details: [email protected]