Surveying the Landscape of Ethics-Focused Design Methods

SHRUTHI SAI CHIVUKULA, Purdue University, USA

ZIQING LI, Purdue University, USA
ANNE C. PIVONKA, Purdue University, USA
JINGNING CHEN, Purdue University, USA
COLIN M. GRAY, Purdue University, USA
arXiv:2102.08909v1 [cs.HC] 17 Feb 2021

Over the past decade, HCI researchers, design researchers, and practitioners have increasingly addressed ethics-focused issues through
a range of theoretical, methodological and pragmatic contributions to the field. While many forms of design knowledge have been
proposed and described, we focus explicitly on knowledge that has been codified as “methods,” which we define as any supports for
everyday work practices of designers. In this paper, we identify, analyze, and map a collection of 63 existing ethics-focused methods
intentionally designed for ethical impact. We present a content analysis, providing a descriptive record of how they operationalize
ethics, their intended audience or context of use, their “core” or “script,” and the means by which these methods are formulated,
articulated, and languaged. Building on these results, we provide an initial definition of ethics-focused methods identifying potential
opportunities for the development of future methods to support design practice and research.

CCS Concepts: • Human-centered computing → Interaction design process and methods; • Social and professional topics
→ Codes of ethics.

Additional Key Words and Phrases: design methods, ethics, values, design practice

Draft: February 11, 2021

1 INTRODUCTION
There is a growing interest in socially responsible design, evidenced by the efforts of practitioners and third-sector
organizations alike in building awareness and support for ethically-centered design practices [8, 17, 61, 67]. These
efforts have frequently built upon academic discourses such as ethics [83, 130, 141], values [56, 61, 64], moral philosophy
[29, 61], and critically-oriented models of participation [26, 97, 130]. While substantial efforts have been made to describe
the value-centered or ethics-focused methods landscape from a scholarly and empirical perspective (e.g, value-sensitive
design [61, 63], values at play [57], ethical standards or codes [78], policies [1, 73]), a specific landscape of ethics-focused
methods that are intended to pragmatically support the actions of designers and technologists in their everyday work
is less well defined.
In prior work, design and HCI researchers have defined and engaged with methods as cognitive and pragmatic
supports [80, 91, 136], a means of encouraging creative production [95], an enabler of dialogue and communication
during design activity [122], and a way of bridging multiple disciplinary ways of knowing to inform effective practice [89].
However, design methods by themselves do not contain any action or inherently prescriptive or binding directives [81],
but rather are tools which enable design activity through the knowledge they contain [79, 91], under the control of the
designer who activates this knowledge in situated and pragmatic ways to support their design activity [79, 105, 136, 137].
Thus, we seek to investigate not only the means of supporting design practices in a broad sense, but also seek to describe
the relationship between knowledge bound up in methods to the activation of that knowledge to create the potential
for socially responsible design practices. The investigation of knowledge in design methods points towards questions,
1
 Chivukula, et al.

such as: How is a method structured? What are the constituent elements of a method? and What is the language used
to describe a method? In parallel, the analysis of methods through an ethics-focused lens reveal yet more questions,
such as: How do methods enable designers to identify and act upon potential social impacts? Can methods guarantee
ethical outcomes? and How are ethical concerns inscribed into the language of methods? We do not seek to answer
these questions in full, yet reveal this landscape of questions to demonstrate the potential broader impact of this work.
In this paper, we identify, analyze, and describe a set of existing ethics-focused methods designed to support design
research and practice for a range of audiences. Building on a content analysis of 63 collected methods, we describe
how these methods operationalize ethics, are framed for particular audience(s), and are built to convey specific types
of knowledge and sensitizing concepts. Across this collection of methods, we have deconstructed the language and
specifications from the method source to identify the intended audience(s), format of guidance, interaction qualities,
utilization of existing knowledge or concepts, implementation opportunities within design processes, the “core” or “script”
of the method, and the ways in which the method builds upon or refers to existing ethical frameworks. These aspects of
ethics-focused methods aid us in characterizing a current landscape of ethical support for practitioners, elucidating
opportunities for the adaptation of existing methods and the creation of future ethics-focused or value-centered methods
for supporting design research and practice.
The contribution of this paper is three-fold: 1) We identify existing ethics-focused or value-centered methods in
order to map the space of current ethical support for designers; 2) We deconstruct the language and specifications
of these methods to describe the framing used for the intended audience(s) and describe the means by which ethics
is operationalized, facilitating more detailed inquiry into how methods are constructed and how they might further
support ethical awareness and action; and 3) We identify opportunities, synergies, and gaps in ethics-focused methods,
providing a roadmap for the creation and adaptation of methods that are resonant with the needs of practitioners.

2 BACKGROUND WORK
2.1 Design Knowledge and Methods
The notion of “design knowledge” has been extensively researched in the design and HCI literature, broadly defining
what constitutes design knowledge (e.g., patterns of reflection [43], ontologies [145]), levels of instigation of design
knowledge in design activity [76, 99], and different types of design knowledge [104, 105]. For the purpose of this paper,
we explore “methods” as a particular form of design knowledge that enables “the creation of design states” [99] that
support and advance a designer’s capability [105]. We draw on the definition given by Stolterman and colleagues [136]
of design methods as “tools, techniques, and approaches that support design activity in [a] way that is appreciated
by practicing interaction designers,” and Gray’s [81] definition that describes design methods as “tool[s] that allow
designers to support thinking, reflecting and acting upon design activities.” Within this framing, we wish to further
describe how methods-focused knowledge allows researchers to better understand design practices, including the
identification of areas where there is stronger and weaker support. Prior research on the use of methods by practitioners
has shown evidence that methods are largely selected and used based on emergent aspects of the design context, where
practitioners leverage knowledge enabled through the use of tools either for thinking or generating artifacts [136]. In
this sense, methods are primarily activated through a “mindset” rather than a precisely defined way to conduct design
activity, and the performance of any given method or combination of methods is dependent on how a designer chooses
to appropriate methods to support their design work [79].

2
Surveying the Landscape of Ethics-Focused Design Methods

We view methods as a form of design knowledge that does not function alone, but is rather activated through the
designer’s activity and judgment, reflecting on the design knowledge contained within the methods [76, 81, 137]. This
knowledge can be abstracted further to describe the repertoire of an individual designer, which includes both stores of
existing design precedent [125] and larger assemblages of tool knowledge that Gray et al. [80] have previously referred
to as a designer’s conceptual repertoire. Thus, in building and elaborating the inherent structures of existing design
methods, we are able to point towards a conceptual repertoire that is implicit in both design knowledge and use. This
notion of a conceptual repertoire also builds on previous work in the HCI and design communities that has interrogated
both prescribed and performative accounts of design practices, including both the exploration and performance of
methods by practitioners from Goodman and colleagues [75, 76] and Reeves [120], and the differentiation between
codification and performance proposed by Gray [81]. In particular, we focus on the notion of method “cores” [79] to
describe an inscribed potential for design moves contained within methods, driven by a synergistic overlap of context,
script (embedded instructions), and the lived experience of the designer.
We also build upon prior work that has defined and curated a range of methods to support design activity, with such
work having a stated goal of describing methods in ways that are simple enough for designers to adapt, apply and
combine different methods in various ways [44]. Löwgren and Stolterman [105] have built upon this notion of method
reuse, stating that methods should be accessible, flexible, and adaptable for designers to apply either independently
or alongside the designer’s current “toolbox” in different contexts. Finally, Stolterman [135] has claimed that any
knowledge introduced into design practice should bear a “rationality resonance,” whereby the content of methods
should resonate with the complexity of practice. Following this guidance and framing, there have been numerous
attempts to create and curate design method collections, including: method for creativity and innovation to improve the
range of design production [30, 113]; a collection of UX evaluation methods [143]; a classification of methods, including
traditional, adapted, innovative methods, and methods for interpretation and analysis [89]; a popular collection of UX
research and design methods for design students and practitioners titled Universal Methods of Design [90]; a design kit
for Human-Centered Design practice by IDEO [6]; a collection of product design methods and approaches known as
the Delft Design Guide [140]; an overview of strategies and methods for design innovation titled Design. Think. Make.
Break. Repeat [139]; and a set of generative approaches for design research [124]. While the list of curated collections of
methods is already substantial, and still growing, we intend to build upon these collections with an explicit focus on
ethically-centered practice, building upon existing language to describe methods while also proposing new vocabulary
to conceptualize, categorize, and propose links within and among methods.

2.2 Supporting Ethical Design Practice

HCI, Science and Technology Studies (STS), and design researchers have previously explored ethical practice across
multiple dimensions, including: theoretical accounts [64, 130], methodological descriptions [57, 61, 107], identification of
pragmatic and practice-led work [17, 40, 82, 83, 129, 134], and philosophical accounts [50, 142]. When focusing on prior
research contributions relating to methodology, we have identified numerous frameworks that propose methodological
means for designers to engage in value discovery and implementation. Common and well known methodologies include
Value Sensitive Design (VSD; [61, 63]) and Values at Play [57]. Other researchers have proposed strategies for designers
or technologists to advocate for values in practice contexts, including organizationally-focused approaches such as
Shilton’s “Value Levers” [129] or van Wynsberghe’s “Ethicist as Designer” [141]. It is claimed that these strategies and
methodologies can “open new conversations about social values and encourage consensus around those values as design
criteria” [129]; identify new ways to expose and reflect upon designers’ responsibility or attitudes towards value-based
3
 Chivukula, et al.

decisions [84, 85]; propose suggestions for critical and reflective technical practice [21]; foreground tools for value
comprehension in particular contexts [61]; provide practitioners with ethical codes for computing work [1]; frame
policies for ethical responsibility for organizations [73]; and offer requirements for ethics curriculum for computing and
engineering education [77, 92]. This range of prior work illustrates the efforts of the HCI, STS, and design communities
towards identifying opportunities for supporting ethically-focused work practices. However, as an additional point of
complication, portions of this prior work has been critiqued regarding its lack of resonance in authentic work settings,
or due to the lack of translation of these practices from academia to practice [83, 107, 130]. In this paper, we focus
our efforts on surveying the landscape of ethics literature through the framing of design methods, with the goal of
gathering and characterizing the existing landscape of ethics-focused and value-conscious design frameworks.

3 OUR APPROACH
To map the landscape of existing ethics-focused methods, we collected a total of 63 methods and conducted a content
analysis [93, 118] to describe the knowledge contained in these methods. This content analysis included the characteri-
zation of these methods on various levels and dimensions that will be detailed below as a part of our analysis approach.
The research questions addressed through this paper are as follows:

(1) What design methods have an ethical focus, and how is ethics operationalized in these methods?
(2) Who are the intended audience(s) for these methods?
(3) How are these methods formulated, articulated, and languaged?

3.1 Researcher Positionality and Rigor

Our approach in using content analysis [118] involved a process of constant reflexivity and researcher alignment,
given the complexity of the method and the ill-defined nature of our topic of interest. All researchers involved in this
process have taken both design and qualitative research methods coursework, and/or were involved in previous research
projects that used content analysis or similar qualitative or critical analysis methods. Additionally, all researchers had
prior experience engaging with conventional design methods through classroom projects and/or professional design
work. These experiences enabled our research team to collect and characterize these methods, as we collectively brought
knowledge of a broad spectrum of ethics-focused knowledge and design expertise. We reflexively engaged in open
and axial coding as a key part of our content analysis process, employing strategies such as coder comments to track
and build consensus, note taking and memoing to create robust coding schemes at every stage, peer debriefing of each
others’ codes to improve the rigor of the analysis process, and regular conversations with the research team to ensure
alignment with generated coding schema at each stage.

3.2 Data Collection

Through a series of structured web searches to locate ethics-focused methods, we collected a list of 89 methods/ tools/
approaches to begin our collection. The searches began by considering the Value Sensitive Design (VSD) methods
[61], which enabled us to characterize the nature and purpose of ethics-focused methods, leading to our web searches
on Google, Google Scholar, and the ACM Digital Library. The following keywords were used for the search queries:
“ethics focused methods,” “ethical tools in design,” “ethics methods,” “HCI ethics and values methods,” and other related
combinations of these terms. The methods we located were considered to be part of our initial collection if they had
a clear ethical valence or had a stated intention to produce ethical or socially-responsible outputs. No specific year
4
Surveying the Landscape of Ethics-Focused Design Methods

ranges were used as filters; nevertheless, we sought to identify as many methods fitting our criteria as possible within
both traditional academic literature and from practitioner sources, given the lack of clear and consistent language to
search using a more traditional “systematic review” approach. We more fully define our inclusion and exclusion criteria
below. The precise scope of “ethical outputs” was not defined until the end of the analysis, as our goal was to identify
methods that broadly had social or human considerations during the design process that related to an ethical valence.
We found methods published only between 2008 and 2020, but our search strategy may have missed methods published
in earlier years, or different terms might have been used to describe such methods. All methods were collected in a
spreadsheet with descriptors such as the title of the method, published year, author names, and source files (documents
or web links). The source files aided us in accessing the method’s description, which was our primary unit of analysis.

3.2.1 Exclusion and Inclusion Criteria. For the purpose of our analysis, we sought to include any design method that
was created with the intent of supporting value-centered, ethically-focused, or socially responsible decision making
practices, as indicated by the method description. We recognized that some methods functioned as methodologies, and
other methods contained multiple sub-methods; in these cases, we sought to identify the smallest method unit for
analysis to increase precision. Through our reflexive data collection and analysis process, we also identified several
exclusion criteria to narrow our focus. First, we excluded any methods that were computationally, algorithmic, and
UI-focused. For example, by computational or algorithmic focused, we refer to toolkits that offer a Python package
to computationally test for biases, and algorithms that are intended to mitigate bias in datasets and models, such as
the AI 360 Fairness Kit[2] that was created for supporting software developer work. Second, we excluded UI-focused
packages such as the IF Data Patterns Catalogue [4], which includes a set of interface choices suggested for handling
user data. Third, we excluded codes of ethics [78, 147] and technology or legal policies [155] as past work has evaluated
the role of these codes in professional practice [37, 100], and the focus of these tools is generally on professional
practice and not specific to design decision making. Fourth, we excluded methods intended to improve accessibility (e.g.,
recommendations for optimizing screen reading) and inclusivity (e.g., general broadening of participation in digital
technologies), since this is already a well-defined area of technology practice and scholarship. Finally, we excluded any
entries that were not clearly expressed as a design method; as an example of the latter type, Stark [133] proposed a
translational means of involving artists for “work to produce a sense of defamiliarization and critical distance from
contemporary digital technologies in their audiences.” Using these inclusion and exclusion criteria, we identified a list of
83 ethics-focused or value-centered methods, tools, approaches, conceptual vocabulary, methodologies, or frameworks.
At this stage, we recognized that we had a heterogeneous collection of theoretical frameworks, concepts, methodologies,
approaches, methods, which led to a further classification effort as described in the following section. We went through
a reflexive process to define various potential classification approaches to define a final set of actionable methods.

Table 1. Classification of Collected Ethics-Focused or Value-Centred methods

Types of Framing Examples

Methods Detailed and described in Figure 1 and 2

Feminist HCI [27, 138], dark patterns [35, 86, 106, 108], Data Feminism [49]
Theoretical Commitments and others [15, 42, 48, 58, 84, 114, 119].
Speculative Design [52], Critical Design [25], Reflective Design [126], and
Conceptual Frames others [51, 94, 101].

5
 Chivukula, et al.

3.2.2 Classification of Collected Artifacts. Building on our collection of 83 artifacts, we sorted them into three main
categories based on their potential function in design activity. As shown listed in Table 1, these functions include:
prescriptive methods, theoretical commitments, and conceptual frames.
Methods: These provide guidance on a practical level, indicating to the designer how they might apply, operationalize,
or activate ethics and values in technology design work. For this paper, we identified a list of 63 ethics-focused or
value-centered methods prescribed for design action, which will be referred to simply as “methods” throughout the
remainder of the paper. We will further elaborate how these methods serve as the main contribution of this paper, and
we focus on this set to answer our research questions.
Theoretical Commitments: These provide guidance to designers on a theoretical level by characterizing the
designer’s ethical commitments (e.g., Data Feminism [49] and Ethical by Design: A Manifesto [114]), listing qualities
required for building ethical outcomes (e.g., Feminist Interaction Design Qualities [27] and dark patterns [35]), describing
existing designs that are manipulative or value-centered (e.g., Asshole designer properties [84] and Nodder’s Seven Sins
[119]), or suggesting organizational structural changes to include ethicists to incorporate ethical reflection into the
product (e.g., Ethicist as Designer [141]). Theoretical commitments do not tell the designer precisely how to engage in
some of these practices or point towards actionable ways of implementing the concepts; these commitments are not yet
procedural in form or defined for the designers in a way that directly activates their principles in concrete contexts, but
rather suggests the required perspectives and language that might be considered when building a prescriptive method.
For example, Bardzell’s Feminist HCI commitment [27] lists qualities that “characterizes feminist interaction” such as
pluralism, participation, advocacy, ecology, embodiment, and self-disclosure; these qualities could be used to create one
or more methods for Feminist Interaction Design by defining steps or other means by which designers could apply these
qualities in their design work. Other examples that fall under theoretical commitments include dark patterns strategies
[17, 86, 106, 108], Design Justice [42], and In-Action Ethics [58].
Between prescriptive methods and theoretical commitments lie methodologies which include a theoretical framing
or umbrella of relevant and appropriate practices that can be applied in a design situation, often without the suggestion
of specific tools and techniques. As one example of a methodology, Values at Play [57] suggests that the user “discover,
analyze, and integrate values” specifically to game design; however, this methodology can be applied across any design
situation within these stages. Other examples that are included as methodologies in our collection include Value Levers
[129], Research through Design Fiction [32], and VSD [61].
Conceptual Frames: These provide guidance at an epistemological level, providing a more expansive set of proposed
practices and knowledge which point towards broader approaches to building knowledge. For instance, a Critical Design
approach [25] focuses on non-affirmative design practices, recognizing the knowledge that is built in the process of
creating design artifacts. Other examples that fall under conceptual frames include Adversarial design [51], Agonistic
design [31], Critical Design [25], Postcolonial computing [94], Reflective Design [126], Speculative design [52], and the
Queering of HCI [101].

3.3 Data Analysis

Using the method descriptions as our unit of analysis, we describe our data analysis procedures in three broad steps, as
guided by Neuendorf’s content analysis approach [118]: 1) familiarizing ourselves with the data set; 2) creating and
validating the coding scheme; and 3) performing open and axial coding.

6
Surveying the Landscape of Ethics-Focused Design Methods

Table 2. Codebook of method characteristics.

Characteristic Description

Primary Audience Intended users of the method.

Discipline/Domain Ex: Engineering, software development, design, etc.

Tangible form in which the method has to be used or structured. Axial
Primary Medium codes include: worksheet, template, cards, document/ guidebook, physical
manipulatives, videos, idea/ practice, and game.
Ontological description and knowledge proposed as a part of the method’s
description. Axial codes include: steps, guidelines, framework, lens/ perspec-
Type of Guidance tive, reflective questions, examples, heuristics, epistemology, methodology, and
case study.
Elements the method operates on, which is inputted by the user(s) of
the method. Axial codes include: user information, design artifacts/ ser-
Input vices, users/stakeholders, values, framing constraints, problem frame, scenar-
ios/context, and research material.
Actions expected from the users while using this method. Axial codes
Mechanics include: altering, storytelling, filtering, creating, mapping, and evaluating.
Tangible results after using this method. Axial codes include: values, con-
Output cepts, research outcomes, evaluation, users/stakeholders, opportunities, proce-
dural information, and research outcomes.
Expectations from the user(s) and the ways in which the output might be
Outcome Expected manipulated by the user(s).
Established design methods/ methodologies that are referenced or used as
Established Method(s) used part of using or building the method.
Established theoretical concepts that are used in this method and the theory
Sensitizing Concepts that has given the method’s vocabulary.
Environmental or logistical aspects of using the method. Axial codes include:
Context of Use group types (team, individual) and ecology types (industry,instructional
context).
Design phase in which the method is prescribed to be used or can be used.
Design Process Steps Axial codes include: a priori phases from Univeral Methods of Design [90].
Ethics theor(ies) inscribed into the method as a part of the “outcome ex-
Ethical Framework(s) pected.” Axial codes include: deontological, consequentialist, virtue, pragma-
tist, and care ethics.

Attributes Characteristics of the method mentioned in the description.

The central mechanic of using this method that remains relatively stable
during adaptation and use. Axial codes include: posture types (eliciting
Core values, critically engaging, defamiliarizing) and actions (consensus building,
evaluating, framing, generating).

7
 Chivukula, et al.

3.3.1 Familiarizing with Data. We began our analysis process by performing a close reading of several methods
to familiarize ourselves with the organization of knowledge and language used to characterize each method. Two
researchers individually identified preliminary codes for nine methods (including a diversity of topics, audience, and
goals), including potential descriptors that aided our team in characterizing the content of the methods, pointing towards
an initial coding scheme. The focus of this content analysis was to describe the method and analyze its characteristics
based on a clear reading of its text, and not the instigation, creation or evaluation of the method in the context of
practice.
As a part of the initial coding scheme, the researchers listed candidate descriptors that ranged from the form of the
method, its potential application in design processes, expected outcomes, intended audience, attributes, and means of
interaction with the method. After multiple rounds of iteration and discussion among the research team, a preliminary
codebook of descriptors was created as detailed in Table 2. The more robust axial codes underneath the broader
descriptors were determined later in the analysis process. During this stage, we also began to identify open codes
[34, 123] to describe potential methods “cores,” alongside researcher-inferred assumptions about the potential primary
audiences that may use the method. By “core” of the method, we refer to the central mechanic of using this method that
remains relatively stable during adaptation and use [79, 81].

3.3.2 Creating and Validating Coding Schemes. During the second stage of analysis, we focused on validating the
overarching descriptors from the initial coding scheme by revisiting the same nine methods coded in the previous
round. We used a linked set of spreadsheets to conduct and document the content analysis of all the collected methods.
This approach (facilitated by the tool, AirTable1 ) aided us in clearly building the audit trail of our coding process and
relating these codes to previous coding work, increasing the validity and robustness of our codebook. At this stage, we
began with an iterative process of open and axial coding under each of the main descriptors as described in our final
stage of analysis. For each descriptor, we reflexively moved through stages of open coding, identification of preliminary
definitions in a codebook, and extended conversation among members of the research team. Through deliberation over
multiple weeks and rounds of coding and revisions to the codebook, we identified a final codebook for each descriptor
set. All codebook elements, and the use of these elements in the coding process, were evaluated by pair coding and all
application of descriptors was discussed until full agreement was reached.

3.3.3 Open and Axial Coding Descriptors. With the high-level descriptors (Table 2) finalized and the researchers aligned
in their understanding, we conducted open coding of the full set of methods using the codebook. Once this initial coding
was completed, we used these open codes to identify axial codes within each descriptor, using the process described
above. The final round of analysis included summative, top-down coding using the final descriptors and sets of axial
codes, including the type of guidance, primary medium, input, mechanics, output, core; all axial codes are listed in
the description column of Table 2 in italics. The role of axial coding varied for each descriptor, and is detailed in the
findings section below.

4 FINDINGS
In this section, we report on the findings of our content analysis, divided by research question. The three main
subsections include: 1) The method’s operationalization of ethics, where we describe the core of the methods and ethical
frameworks that are activated; 2) The intended audience for these methods; and 3) The formulation, articulation, and

1 https://airtable.com

8
Surveying the Landscape of Ethics-Focused Design Methods

conceptual language used to describe these ethics-focused methods. A summary of the method descriptors is provided
in Figures 1 and 2.

4.1 RQ#1: Operationalization of Ethics

In this subsection, we identify how ethics or values were operationalized in these methods. We describe this op-
erationalization through two properties: 1) the core of the method and 2) the ethical framework(s) activated in the
method.

4.1.1 Method Core. By method core, we refer to what we inferred as the central conceit or essence of the mechanics
of the method. Each method’s core was identified from two groups: 1) Postures: eliciting values (n=32), critically
engaging (n=10), defamiliarising (n=21), and 2) Actions: consensus building (n=2), evaluating (n=26), framing (n=20)
and generating (n=15). We propose “postures” to be very specific to ethics-focused or value-centered methods, whereas
the “actions” apply across any design method. Cores involving postures target attitudes towards a certain action, either
to identify an existing or generated list of values as a conceptual frame (eliciting values), engage in critical perspectives
or theories drawn from critical theory as an epistemological argument (critically engaging), or take part in alternative
forms of looking at existing concepts or forms of thinking (defamiliarising). Cores involving actions encourage users to
align their decision making with other stakeholders (consensus building), assess and validate the decision (evaluating),
map the design space for using the method (framing), and produce design artifacts (generating). For example, the
Inclusive Design Toolkit [10] had a core of “eliciting values,” since it proposes to evaluate a design artifact using values of
inclusivity or accessibility. In Judgment Call the Game [24], the method core focuses on “defamiliarizing,” through which
designers can “evaluate” a design scenario through reviews and ratings from the perspective of alternative users in the
situation. Another example is Security Fictions [110], which asks users of the method to “defamiliarize” themselves to
think differently about security issues as they “generate” concepts to solve security threats. As these examples illustrate,
the two groups of cores represent how ethics is operationalized based on the postures leading to those actions.

4.1.2 Activated Ethics Framework(s). We identified how each method related to established ethical frameworks as a way
to illustrate and provide us a vocabulary for how ethics were operationalized; drawing from multiple key texts in the
philosophy literature, including Becker [28], Kant [96], Aristotle [23], and Gert [66]. We used the following frameworks
to categorize the methods: deontological ethics (n=11), consequentialist ethics (n=39), virtue ethics (n=8), pragmatist
ethics (n=40), and care ethics (n= 1). Table 3 summarizes the ethical framework(s) and associated methods. Methods
that used a deontological ethical framework manifest certain values through the method as a “duty” of the user to
implement through their decision making. For example, Ethical Contract [68] requested that the users divide the ethical
responsibilities in the project planning stage and physically sign the document (provided by the method) as a means of
foregrounding their duty in accepting ethical responsibility. Methods that used a consequentialist ethical framework
focus primarily on evaluating or considering consequences of the decision made. For example, Re-shape [127] requested
that students reflect on their actions towards physical movement data collection and its impact on the community
through which they were navigating. Re-shape was also the only method that relied upon a care ethics framework to
foreground a reciprocity of care towards the community through data. Methods using a virtue ethics framework focused
its impact on the designer themself, seeking to build their ethical awareness. For example, HuValue [98] focused on
building students’ sensitivity towards human values by engaging with its knowledge in their design activities. Methods
using a pragmatist ethical framework considered the designer’s judgment, situationality, and context of decision making
as a landscape through which to consider ethical complexity. For example, Moral Value Map [71] asked the designer
9
 Chivukula, et al.

Method Name Medium Context Design Phase Core Input Mechanics Output

Consensus Building
Critically Engaging

Defamiliarising
Eliciting Values
Instructional

Generating
Evaluating
Individual

Framing
Industry
Team
360 Reviewa 1 2 3 4 5   Users/ Stakeholders Mapping Opportunities

Adversary Personas 1 2 3 4 5   User Information Storytelling User Information

Blackmirror Brainstorming 1 2 3 4 5   Scenarios/Context Storytelling,Creating Evaluation Results

Co-evolve Technology... c 1 2 3 4 5   Problem Frame Creating Opportunities

Data Ethics Canvas 1 2 3 4 5   Scenarios/Context Altering,Evaluating,Mapping Opportunities,Procedural Information

De-scription b 1 2 3 4 5   Design Artifact Altering,Mapping Values

Design Fiction Memos 1 2 3 4 5   Scenarios/Context,Data Storytelling Values,Opportunities

Design for Social Acc... 1 2 3 4 5   Problem Frame Creating,Mapping,Evaluating Concepts

Design with Intent 1 2 3 4 5    Constraints Creating,Evaluating,Filtering Concepts

Dichotomy Mappinga 1 2 3 4 5   Design Artifact Evaluating Evaluation Results,Opportunities

Diverse Voices c 1 2 3 4 5   Research Material Storytelling Evaluation Results

Eliciting Values Refl... 1 2 3 4 5   Scenarios/Context Storytelling,Creating,Evaluating Values

Envisioning Cards c 1 2 3 4 5   Design Artifact Storytelling,Filtering Concepts,Values

Ethical Contractb 1 2 3 4 5   Design Artifact Altering,Storytelling Values

Ethical Disclaimerb 1 2 3 4 5   Scenarios/Context Altering,Mapping Opportunities,Values

Ethicography 1 2 3 4 5   Research Material Mapping Concepts

Ethics Canvas 1 2 3 4 5   Design Artifact Altering,Storytelling,Mapping Users/Stakeholders,Values,Procedural Information

Ethnographically Inform... c 1 2 3 4 5   Constraints Creating Research Outcomes

GenderMag 1 2 3 4 5   Design Artifact Storytelling,Filtering,Evaluating Evaluation Results,User Information,Opportunities

Hippocratic Oath a 1 2 3 4 5   Values Creating,Storytelling Values,Procedural Information

HuValue 1 2 3 4 5   Problem Frame Filtering,Mapping,Storytelling,Evaluating Concepts,Evaluation Results,Values

Idea Generation... 1 2 3 4 5   User Information Mapping,Storytelling,Creating Concepts

Inclusive Design Toolkit 1 2 3 4 5   Design Artifact Evaluating,Mapping Opportunities,Evaluation Results

Inverted Behavior Model a 1 2 3 4 5   Design Artifact Evaluating Opportunities,Evaluation Results

Judgment Call the Game 1 2 3 4 5   Design Artifact Storytelling,Altering,Filtering Users/Stakeholders,User Information,Evaluation

Layers of Effecta 1 2 3 4 5   Design Artifact Evaluating,Mapping Opportunities

Make It Critical 1 2 3 4 5   Scenarios/Context Filtering,Mapping,Creating Concepts

Making an Ethical Decision 1 2 3 4 5   Users/Stakeholders Filtering,Mapping,Evaluating Evaluation Results

Maslow Mirroreda 1 2 3 4 5   Design Artifact,Users/Stakeholders Evaluating Evaluation Results,Opportunities

Metaphor Cardsc 1 2 3 4 5   Scenarios/Context Storytelling,Creating Concepts,Opportunities

Microsoft Inclusive Design... 1 2 3 4 5   Constraints Creating Concepts,Opportunities

Medium Legend: Design Phase Legend:

Cards Worksheet 1 Planning, Scoping, & Definition 4 Evaluation, Refinement, & Production

Document/Guidebook Physical Manipulative 2 Exploration, Synthesis, & Design Implications 5 Launch & Monitor

Template Video 3 Concept Generation & Early Prototype Generation

a
Part of the Design Ethically Toolkit.
b
Part of the Ethics for Designers Toolkit.
c
Part of the Value Sensitive Design Toolkit.

10
Fig. 1. Methods (part 1) classified and organized by medium, context, design phase, core, input, mechanic(s), and output(s).
Surveying the Landscape of Ethics-Focused Design Methods

Method Name Medium Context Design Phase Core Input Mechanics Output

Consensus Building
Critically Engaging

Defamiliarising
Eliciting Values
Instructional

Generating
Evaluating
Individual

Framing
Industry
Team
Model for Informed ...c 1 2 3 4 5  [...] [...] Values

Monitoring Checklista 1 2 3 4 5   Design Artifact Evaluating Evaluation Results,Opportunities

Moral Agent b 1 2 3 4 5   Constraints Filtering,Creating,Storytelling Concepts

Moral and Legal Decks 1 2 3 4 5   Design Artifact Evaluating,Filtering Evaluation Results,Values

Moral Value Mapb 1 2 3 4 5   Design Artifact Filtering,Altering Values

Motivation Matrixa 1 2 3 4 5   User Information Storytelling User Information,Opportunities

Multi-lifespan Co-design c 1 2 3 4 5   Problem Frame Creating Research Outcomes

Mutli-lifespan Timelinec 1 2 3 4 5   Design Artifact Mapping Opportunities

Normative Design Sche...b 1 2 3 4 5   Problem Frame Altering,Evaluating Concepts,Opportunities

Re-Shape 1 2 3 4 5   Data Mapping,Evaluating Evaluation Results

Scalable Assessments...c 1 2 3 4 5   Design Artifact Storytelling,Evaluating Values,Evaluation Results

Scenario Co-Creation 1 2 3 4 5   Scenarios/Context Storytelling,Filtering Opportunities,User Information

Security Cards c 1 2 3 4 5   Design Artifact Filtering,Mapping Opportunities

Security Fictions 1 2 3 4 5   Problem Frame Storytelling Concepts

Speculative Enactments 1 2 3 4 5   Scenarios/Context Storytelling Research Outcomes

Spotify Design Investig... 1 2 3 4 5   Design Artifact Altering,Evaluating Opportunities

Stakeholder Analysis c 1 2 3 4 5   Users/Stakeholders Creating,Mapping Users/ Stakeholders

Stakeholder Tokens c 1 2 3 4 5   Constraints Storytelling,Mapping Users/ Stakeholders,Values

Tarot Cards of Tech 1 2 3 4 5   Design Artifact Filtering,Evaluating Opportunities

The Ethical Design Score... 1 2 3 4 5   Design Artifact Altering Evaluation Results

The Ethics and Inclusion... 1 2 3 4 5   Design Artifact Filtering,Evaluating Procedural Information

The Oracle for Transfem... 1 2 3 4 5   [...] Creating,Filtering Concepts

Timelines 1 2 3 4 5   Design Artifact Storytelling,Filtering Values,Opportunities

Value Dams and Flows c

– 1 2 3 4 5   Design Artifact Evaluating Values,Evaluation Results

Value Source Analysisc 1 2 3 4 5   Problem Frame Mapping Values

Value-oriented Codingc 1 2 3 4 5   Research Material Mapping Research Outcomes,Values

Value-oriented Mock-up... c 1 2 3 4 5   Constraints Creating Concepts,Values

Value-oriented Semi-...c 1 2 3 4 5   Problem Frame Storytelling Research Outcomes

White Hat Design Patterns 1 2 3 4 5   Constraints Evaluating,Creating Concepts

Fig. 2. Methods (part 2) classified and organized by medium, context, design phase, core, input, mechanic(s), and output(s).

to choose relevant human values relevant to “your design” and combine these values with the “context of use” at the
designer’s discretion, encouraging flexible use of the method.
11
 Chivukula, et al.

Table 3. Ethical frameworks by method

Methods Relying upon this

Ethical Theory Methods Using this Theory. . .
Framework
. . . foreground particular values through the
[10–12, 60, 67, 68, 70, 88, 98, 103,
Deontological Ethics method as a “duty” of the practitioner to im-
131, 132]
plement or address in their decision making.
[3, 10, 14, 16, 18–20, 24, 36, 47,
53, 55, 60, 69, 71, 88, 102, 109–
. . . focus on evaluating or considering conse-
Consequential Ethics 111, 115, 117, 121, 127, 144, 148–
quences of design decisions.
150, 153, 154, 156–158, 160–
163]
. . . address the designer themself, providing
Virtue Ethics [19, 24, 33, 54, 59, 98, 128, 159]
guidance to increase their ethical awareness.
[3, 11, 13, 14, 16, 20, 22, 33, 36,
. . . consider the designer’s judgment, situation-
39, 45, 54, 55, 59, 62, 65, 68–71,
ality, and the context of decision making as it
Pragmatist Ethics 88, 102, 103, 110–112, 115, 116,
encourages the designer to decide which ethical
127, 144, 146, 150, 151, 154, 160,
lens best addresses the situation at hand.
162, 163]
. . . foreground a reciprocity of care towards oth-
Care Ethics [127]
ers or a community.

4.2 RQ#2: Intended Audience

In this subsection, we describe the implied audience and context for these methods across the following descriptors:
1) the primary intended audience of the methods; 2) the defined context of use of these methods; and 3) the published
format as a means of disseminating the methods.
Primary audience. We relied upon the method developer’s identification of the intended audience from the method
description to infer the primary audience of the method. The stated audience types included: educators (n=3), academic
researchers (n=15), students (n=6), industry practitioners (n=54), or anyone (n=1). Of these methods, 13 did not explicitly
state the primary audience (leaving the researchers to infer the audience), while the remaining methods mentioned their
intended audience in the method description. For methods that intended to encourage conversations and collaborations
among industry practitioners, sub-audiences primarily targeted design professionals (n=28), technology professionals
(n=17), industry researchers (n=12), policy makers (n=2), and managers (n=1). The majority of methods had an intended
audience from only one category rather a combination of multiple stakeholders.
Context of use. We describe the context in which the method is intended to be used within two categories. First, we
sought to identify whether a team and/or individual was the ideal group size for the method to be used. Methods that
appeared to be designed for a group as the intended audience were the most common (n=35), while individuals were a
minority (n=4); methods identified for use by either a group or an individual were coded under both categories (n=23).
Second, we sought to describe whether industry and/or instructional settings were the primary ecological setting for
methods to be used. Methods were coded to represent the anticipated ecological setting, including industry work for
practitioners (n=45), an instructional setting for students and educators (n=3), or research in an academic context (n=4).
A minority of methods (n=9) anticipated use both in industry and academia.
Published format. We identified the published format for each method to identify its dissemination strategy or
availability, thus revealing assumptions regarding the intended audience or the type of knowledge building the method
12
Surveying the Landscape of Ethics-Focused Design Methods

represented. The publication formats we identified included academic papers (n=28), websites (n=20), blog posts (n=16),
books (n=4), and unpublished (n=1; [132]). It is interesting to note that the majority of the methods we analyzed were
published as academic papers, while the primary intended audience of these methods were industry practitioners. The
limited availability of these papers behind a paywall perhaps brings into question the accessibility of these methods for
the intended audience.

4.3 RQ#3: Formulation, Articulation, and Language

In this subsection, we describe and characterize the collection of methods in three ways: 1) Formulation of the methods
as scripted to define input-mechanics-output, existing frameworks/structure and design process implementation of
these methods; 2) Articulation of these methods to the audience to describe type of guidance and medium of these
methods; and 3) Language that formed core of these methods. This section answers our research question #3.

4.3.1 Formulation of Methods. We coded the “script” of these methods which illustrate the structure and interaction
with these methods. We describe the formulation of the methods through major categories: 1) input required, mechanics
of interaction and output generated from the method; 2) existing frameworks or methods used to build the method; and
3) practical implementation of the method in a design process. We present different axial codes, definitions and examples
in the paragraphs below.
Input–Mechanics–Output. The input–mechanics–output sequence describes how the methods are formulated,
pointing towards potential patterns of performance. We identified ten salient inputs, six action-oriented mechanics,
and eight tangible outputs across the collection of methods. We will describe each element of the sequence separately
below using a variety of methods as supporting examples. We have observed that despite a similar input, the change in
mechanic has the potential to result in different outputs, giving us an opportunity to explore the interactions among
these three elements. We provide more details regarding the patterns of interaction in the discussion section.
Input. We coded the required materials or knowledge the method developer wants the users to input as a means
to proceed with the method. The identified inputs include design artifact/service/business (n=23), research material
(n=3), problem frame (n=8), constraints (n=9), users/stakeholders (n=4), user information (n=3), scenarios/context
(n=10), values (n=1), data (n=2), and one method without any required input. Each method may include a wide range of
inputs, but we chose to exclusively code only the most salient input required for each method to provide a more precise
set of entry points. Methods with the input of design artifact/service/business require the user to select an existing
design material, product, or business service. For example, Multi Lifespan Timeline [153] requires the user to provide a
technological design artifact as an input in order to map how this artifact would exist in a social context at different
timelines beyond the product lifecycle. Methods with research materials as an input require users to bring materials
constructed for research purposes, such as interview protocols, co-design materials, tech policy documents, and others.
For example, Scenario Co-Creation Cards [22] require an interview protocol along with the cards for conducting
value-eliciting interviews with a culturally-diverse population. Other examples in this category include the Value
Sensitive Action-Reflection Model [154], which requires the input of co-design materials, and Diverse Voices [7], which
requires a tech policy document for expert panel discussions. Some methods required users to input their problem
frame in order to construct, define, and approach a design space through the lens of that method. We differentiate
this input from constraints based on their level of definition; whereas constraints are expected to be precise, problem
frames are frequently more open-ended. Constraints include stakeholder requirements, a project brief, time constraints,
limited resources, or other explicit requirements that must be met. Methods that require users/stakeholders or user
13
 Chivukula, et al.

information as inputs encourage the description of stakeholder needs in relation to method-guided decision making.
Users/stakeholders indicates who the design is created or evaluated on behalf of, or who needs to be considered in
decision making, whereas user information describes user needs, user actions, and user values (e.g., a persona or user
story). Methods with scenarios/context as an input require a design situation or product scenario that the team has
encountered, or a fictional situation that the user envisions. For example, Data Ethics Canvas [3] requires the user to
formulate a scenario for which they need to plan data collection, storage, or opportunities for analysis. Methods with
values as an input require the user to formulate a list of personal, social, team, company, and/or project values to frame
their decision making. Methods with data as input, for example Re-Shape [128], require the users to provide data to
teach data ethics and use the resulting data as a starting point for analysis and reflection. In a rare example, The Oracle
for Transfeminist Technologies [14] did not require any input to use the method, because the first step of the using the
method involves filtering cards to create a problem space to generate futuristic concepts.
Mechanics. We coded the action(s) expected from the users while using the method as its mechanic. The mechanics we
identified include: altering (n=10), creating (n=19), mapping (n=21), storytelling (n=23), filtering (n=17), and evaluating
(n=23). Depending on the type of guidance provided by the method, there could be more than one type of mechanic
for each method, hence these were non-exclusively coded. Methods with altering as a mechanic expect users to edit a
given worksheet as they follow the prescribed steps/guidelines in the method. For example, the Ethics Canvas [121]
provides a template for the users to collaboratively edit, move and add Post-It notes in appropriate sections to fill out
the template. Methods using creating as a mechanic encouraged a more conceptual and divergent approach whereby
users produce artifacts through brainstorming, sketching, prototyping, and developing as they interact with the method,
instead of providing existing artifacts for users to alter. Methods using mapping expect users to draw connections or
associations between method elements and artifacts created through the method. For example, Ethicography [39] is the
method of value discovery that requires researchers to physically draw links that visualize the conversation change
and growth through a design discussion; Value Source Analysis [33] requires users to identify disagreements among
stakeholders in order to conceptually map values for “other environments.” Methods engaging in storytelling involve an
act of role-playing, narrating stories, performing activities, or playing games as the users interact with the method. For
example, in Judgement Call the Game [24], users role-play as stakeholders and write a fictional review framed up by
a combination of the rating card, stakeholder card, and ethical principle card that the player draws. Methods which
require filtering expect users to select scenarios, stakeholders, or draw cards, selecting salient options from a list of
possibilities either provided by the method or created through the method. Methods with mechanics of evaluating
request users to assess the components provided by the method or artifacts produced through the method. For example,
the Moral and Legal IT Deck [18] provides a wide range of critical questions, legal principles, and ethical principles for
designers to follow and thereby evaluate ethically-related risks of the proposed new technology.
Output. We coded the methods based on the tangible outcome that would be produced when using the method as
the output. The identified outputs include concepts (n=17), opportunities (n=23), evaluation results (n=16), values (n=20),
users/stakeholders (n=4), user information (n=6), procedural information (n=4), and research outcomes (n=6). The
majority of the outputs align with the list of inputs, although there is a clear change in their function. Depending on the
method, it is possible that there is more than one possible or likely output for each methods, hence we non-exclusively
coded for this descriptor. Methods with concepts as an output result in new ideas, sketches, artifacts, and/or inspiration
for future work. Methods resulting in opportunities aid the user in locating ethical risks, recognizing future design
possibilities through the method. For example, Ethical Disclaimer [69] is meant to allow the users to “discuss for which
of the unethical situations you will take responsibility.” Methods with evaluation results as outputs allow the user to
14
Surveying the Landscape of Ethics-Focused Design Methods

assess their design through quantitative scores, ethical scores, design requirements, reflections, or other evaluation
metrics. For instance, to illustrate the range of evaluation results, Ethical Design Scorecards [55] provide an “ethical
score” for the users to indicate the potential ethical valence of their design decision, and Re-Shape [128] allows computer
science students to evaluate their own decision making in the form of a reflection of their responsibilities towards data.
Methods with values as an output define a new mindset or outlook on what elements or abstract principles are most
important in the users’ design process. Methods with outputs such as users/stakeholders and user information have a
similar definition as when they are used as inputs; however, as outputs the information about users is realized and
produced through the methods. An output of procedural information encompasses relevant and possible next steps and
a future plan of action for decision making. Finally, methods with research outcomes include artifacts produced using
the method that are possible sources of future research or analysis, such as design research artifacts (e.g., user stories in
speculative enactments [53]) or co-design materials (e.g., in the Value sensitive action-reflection model [154]).

Patterns of Input->Mechanics->Output. Based on our analysis, we have recorded the number of occurrences of
each possible combination of input, mechanics, and output. As part of this approach, we mapped the interactions or
patterns from “input” of the method as it was exclusively coded and present the most salient and frequently occurring
interaction patterns. These interaction patterns aid us in describing the most typical ways in which the methods function
as specification, and these patterns also elucidate possible opportunities for new or altered methods beyond these
existing interaction patterns. We identified six common interaction patterns and provide their descriptions as follows:

• Design Artifacts->Evaluating->Values/Evaluation Results: This interaction pattern was found in methods that guide
users to evaluate existing design artifacts, resulting in a range of evaluation results. Such methods are intended
to address existing product deficiencies, reveal ethical dilemmas of the system, and discover new values to be
embedded in the design. Methods using this pattern include: Value dams and flows [112], Scalable assessments of
information dimensions [115], Moral and Legal Decks [18], Inclusive Design Toolkit [10], and GenderMag [36].
• Design Artifacts->Mapping->Opportunities: This interaction pattern was found frequently in methods that help
users recombine, envision, and derive new design opportunities from existing artifacts by mapping out method
elements or design space. Methods using this pattern include: Security Cards [47], Mutli-lifespan timeline [153],
and the Inclusive Design Toolkit [10].
• Design Artifacts->Storytelling->Values: This interaction pattern is identified in methods which expect the user
to interact with their design artifacts through storytelling or by playing games in order to explore, elicit, and
engage with values in designed artifacts. Methods using this pattern include: Scalable assessments of information
dimensions [115], Ethics Canvas [121], Ethical Contract [68], and Envisioning cards [60].
• Constraints->Creating->Concepts: This interaction pattern is found in methods with design constraints such as a
design prompt, business timeline, or resource constraints, which results in creating original or iterated concepts.
Methods using this pattern include: White Hat UX Patterns [54], Value-oriented mock-up, prototype, or field
deployment [154], Value Sensitive Action-Reflection Model [154], Moral Agent [70], and Design with Intent
[102].
• Problem Frame->Creating->Concepts: This interaction pattern is used in methods that assist participants in
generating concepts within a given or defined problem frame. Methods using this pattern include: Design for
Social Accessibility Method Cards [131].
15
 Chivukula, et al.

• Scenario/Context->Creating->Concepts: This interaction pattern is seen in methods which results in design concepts
created within a defined, assumed or fictional scenario/context. Methods using this pattern include: Value Sketch
[146], Metaphor Cards [103], and Make It Critical [132].

Existing frameworks/methods used. We sought to identify any existing design methods or frameworks that
methods were built on, relied upon, or referenced. These frameworks are not translated into the method for the user,
but rather they require the user to directly interact with these frameworks in order to successfully implement the
ethics-focused method in their work. We identified two kinds of existing frameworks that were used: 1) established
design methods; and 2) other standalone methods. Not all methods used existing frameworks, with only 25 out of
the 63 methods representing this behavior. Established design methods that were referenced (as listed in the Universal
Methods of Design [90]) included: personas [36, 111], cognitive walkthrough [88], stakeholder map [152], scenarios
[22, 117, 149], experience mapping [132], cultural probes [154], qualitative research interviews (used as required
technique in [22, 45, 115]), and ethnography (used as a basic methodology for Ethnographically Informed [116]). These
methods build upon the familiarity of existing methods or approaches, facilitating the use of these ethics-focused
methods with little prior preparation and expert knowledge. Other standalone methods include less common methods,
and frequently ethics-focused methods, with their own mechanics that are used in one of the methods we analyzed.
Examples include: Ethics Canvas (another ethics focused method [121], which was used to build Data Ethics Canvas
[3]); Design Heuristics (a card deck [5] which was used in the cognitive walkthrough approach in the Idea Generation
through Empathy method [88]); Ethical Disclaimer (another ethics-focused method which was used as an input in Ethical
Contract [68]); Linkography (used as a baseline framework [74] in Ethicography [39]); and a combination of Value
Scenarios [117], Envisioning Cards [60], and Value Sketch [146] (used to build and follow steps in the Value-Sensitive
Action-Reflection Model [154]). These methods require user(s)’ existing knowledge in completing the method, including
knowledge about the functioning of connecting methods or other related methods for using the ethics-focused method.

Design Process Implementation. We coded each method’s suggested use or implementation across existing
notions of design process stages (Table 4). We used an a priori list of five design phases as suggested in Universal
Methods of Design [90] as an existing acknowledged mapping of design methods and process. We have chosen these
five phases as they allow our analysis to build upon connections to implementation of the ethics-focused methods and
the established mappings identified in [90]. Methods identified within Phase 1 included activities as planning, scoping,
and definition, “where project parameters are explored and defined” (n=9). Methods in Phase 2 included activities
such as “exploration, synthesis, and design implications which are characterized by immersive research and design
ethnography leading to design implications” (n=40). Methods in Phase 3 included activities as “concept generation and
early prototype iteration, often involving generative and participatory design activities” (n=27). Methods in Phase 4
included activities as “evaluation, refinement, and production based on iterative testing and feedback” (n=34). Finally,
methods in Phase 5 included activities as “launch[ing] and monitor[ing] the quality assurance testing of design to
ensure readiness for market and public use, and ongoing review and analysis” (n=7).
Based on our analysis, we found the majority of the methods were designed for Phases 2, 3, and 4, with only rare
examples in Phases 1 and 5. For methods coded as Phase 2, the focus was primarily on identifying design implications
which aided the user in framing the problem space in more ethical ways; in contrast, methods coded as Phase 4
encouraged the user to build upon a generated design concept in more ethically-centered ways. As examples of Phase
2-focused methods, card decks were used in Adversary Personas [111] to list potential adversaries in a particular design
16
Surveying the Landscape of Ethics-Focused Design Methods

Table 4. Methods and Design Process Implementation

Design Process Phase [90] Methods

Phase 1 (planning, scoping, and definition) [3, 68, 69, 112, 141, 148, 156, 157, 159, 163]
[3, 7, 13, 18, 22, 39, 45, 47, 53, 59, 60, 65, 67, 70, 72, 98,
Phase 2 (exploration, synthesis, and design implica-
102, 103, 109–112, 115–117, 127, 132, 144, 148, 150, 152–
tions)
154, 156, 157, 159, 160, 163]
Phase 3 (concept generation and early prototype itera- [3, 13, 14, 33, 36, 47, 54, 55, 60, 70, 88, 98, 102, 103, 109,
tion) 110, 131, 132, 146, 148, 149, 153, 154, 156–158]
[10, 11, 13, 16, 18–20, 24, 33, 36, 45, 53–55, 71, 72, 102,
Phase 4 (evaluation, refinement, and production) 110, 112, 115, 116, 121, 127, 144, 148, 149, 153, 156–158,
160–162]
Phase 5 (launching and monitoring) [36, 53, 127, 148, 156, 157, 162]

situation, while Envisioning Cards [60] were used to expand potential issues in the “immediate context of use,” with the
goal of envisioning the potential long-term impact of technology. The actions supported through these methods are
likely to occur before concept generation, with the goal of framing the problem space by providing new ways of viewing
the context. In Phase 3 and Phase 4, methods enable the production of ethically-focused designs and the evaluation of
created or existing designs, respectively. For example, the Design for Social Accessibility Method Cards [131] provide
users with “concrete and real-life scenarios” in Phase 3 to “to generate accessible designs and appropriately engage deaf
and hard-of-hearing users to incorporate social considerations.” In Phase 4, the majority of methods provided ways
for the user to evaluate their decisions or design outcomes, using guidance to refine their decisions. For example, the
Ethics and Inclusion Framework [19] aids the user in calculating the “degree of inclusion of your product or service” or
facilitates “assessment of potential negative outcomes” for intended or unintended stakeholders.

4.3.2 Articulation. In this section, we describe the methods based on the way they are articulated to their respective
audiences. We describe the ways methods are communicated to these audience(s) through two properties: 1) the type of
guidance; and 2) the medium through which the method is communicated.

Type of guidance We coded the descriptors that communicate the scaffolding or means of support to engage with
the method in a way that is accessible to users as type of guidance and the tangible form in which the guidance was
provided as medium. The type of guidance frames how the method is structured and conveyed to the user as instructional
support or scaffolding [46] in the following ways: steps (n=30), guidelines (n=23), framework (n=19), lens/perspectives
(n=14), reflective questions (n=10), examples (n=22), heuristics (n=4), and case study (n=9). The method descriptions
provided in the cited material frequently consisted of more than one kind of guidance, given the structure of the method,
resulting in non-exclusive coding. If the method had multiple components or sub-components, we coded for all kinds
of guidance provided, including any sub-structures of the method. Steps are prescribed instructions to be followed
by the user in order to interacting with the method in the provided order, whereas guidelines do not insist on being
followed in a specified order. For example, Diverse Voices [7] provides “main steps” to be followed by the user starting
with “Select a tech policy document” and additional guidelines under each step to describe how and what kinds of tech
policies documents to be selected. A framework is a defined structure provided by the method developer in the form of
a table, illustration, or schema. Lens/Perspectives are possible attitudes or perspectives provided to focus the thought
processes of the method user, while heuristics are techniques that can be implemented non-deterministically in order to
17
 Chivukula, et al.

guide the user of the method. For example, White Hat UX Patterns [54] lists a set of heuristics “to ensure ethical design”
outcomes, guided by heuristics such as: “Use data to improve the human experience,’ ’ and “Advertising without tracking.”
Reflective questions are posed as questions for the user to critically think through the “input” as intended by the “core”
of the method. Examples are real world scenarios and/or visually illustrated guidance provided along with other types
of guidance, while case studies represent a real world design context through which the method is described rather than
a standalone description of the method. Many of the methods proposed as part of the VSD methodology [61] included
case study-focused guidance that was represented as bound within a specific design decision, taking on characteristics
of a case study.

Medium. We coded the descriptors that describe the tangible form of the methods as its primary medium, which
aided in communicating the above listed types of guidance to the user. The medium also inscribes how the method can
be interacted with by the user in digital or physical format. The different medium types include: worksheets (n=17),
templates (n=17), cards (n= 15), document/guidebook (n= 33), physical manipulatives (n=3), videos (n=1), and games
(n=2). Worksheets are documents where the user is asked to add specified information as they are working with the
method, whereas a template is a document that is expected to be used as a baseline reference in order to interact with
using other components of the method. For example, HuValue [98] has a template with different value groups sectioned
as a part of a circle which is intended to be used as a base with which to “filter” cards of user’s choice under the value
groups. Other physical media include cards or physical manipulatives, while digital media also include videos, and
some methods could be presented in a combination of digital and physical forms. Design with Intent [102] presents
different “lens” through which design artifacts can be evaluated in the form of deck of physical color-coded cards. A
document/guidebook could include a digital or physical standalone booklet that contains a method description and
type of guidance for the user to refer as they are using the method during their design activity. Two methods—Moral
Agent [70] and Judgement Call the Game [24]—were designed to encourage interactions in the form of a board game,
consisting of a combination of several of the media described above. As another example of hybrid media, Moral Agent
[70] consisted of a card deck to draw from, worksheets to write on, and two documents/guidebooks to filter values and
describe the rules of the game.

4.3.3 Language of Existing methods. We coded the descriptors that frame the language of these methods as sensitizing
concepts, drawing on a term by the same name that is often used to identify structure and conceptual foundations in
grounded theory research [38]. These concepts provide a conceptual and methodological vocabulary that the method
developers use to define the expected purpose or core of the method. Depending on the method focus, this vocabulary
ranges from established social constructs (e.g., culture, gender); published policies (e.g., GDPR, EU Draft e-Privacy
Regulation 2017); defined methodologies (e.g., critical design, VSD, speculative design, design fiction, co-design); known
ethical or privacy concerns (e.g., user behavior change, cyber-security, data privacy, data ethics); defined human values
(e.g., privacy, security, spirituality); commonly used interaction design concepts or methods (e.g., form, function, empathy,
scenarios); and applied ethical concepts (e.g., justice, human rights, common good, utility). For example, GenderMag
[9, 36] relies upon the social category of “gender” as a sensitizing concept to frame the designer’s construction of
personas; this social construct shapes the method’s purpose in engaging software developers in a more inclusive form
of building technological artifacts and links the use of the method to broader social and academic conceptions of gender.
The Moral and Legal Decks [18] cards used vocabulary from “relevant rights, principles, definitions and responsibilities
within the: EU General Data Protection Regulation 2016; EU Draft e-Privacy Regulation 2017; EU Network and Information
18
Surveying the Landscape of Ethics-Focused Design Methods

Security Directive 2016; Cybercrime Convention 2001; and Attacks Against Information Systems Directive 2013” to design
the content and guidelines provided through the method, thereby grounding design activity in legal definitions of
privacy and data protection.
These sensitizing concepts had different functions in different methods, with some concepts being used across
multiple methods in different ways to encourage or foreground specific mechanics, design judgments, or framings
of design activity. For example, design fiction was defined as the means by which the designer should create “design
concepts” in the Security Fictions method [110], whereas design fiction was used as an ideology with which the user
could evaluate design concepts in Judgment Call the Game [24]. Design fiction was also treated as an opportunity
space through which to explore and elicit values in relation to privacy in future technological artifacts in Eliciting
Values Reflections method [149]. Overall, we identified more than 80 sensitizing concepts across the set of methods
we analyzed, and few sensitizing concepts appeared to be used consistently. Several key sensitizing concepts which
did appear in multiple methods include: GDPR [18, 55], co-design [53, 146, 153, 154], VSD (all methods within the
VSD methodology), speculative design [53, 110, 132, 149], human values [22, 33, 39, 45, 98, 146, 154] and design fiction
[24, 53, 110]. This analysis demonstrates that most methods include their own distinct vocabulary which is generally
not shared or standardized across multiple methods, illustrating both the variety in the existing ethics-focused methods
and the lack of consistency across methods.

5 DISCUSSION
In this paper, we have identified a range of descriptors of ethics-focused methods that allow us to identify existing
mechanisms for ethical support, along with opportunities for the development, adaptation, and dissemination of new
methods. In this section, we outline how these findings point towards gaps in our current knowledge of ethics-focused
methods, identifying spaces for new method development, and also revealing ways to interrogate the performance and
performativity of these methods in complex organizational contexts. We conclude this section with a more philosophical
discussion regarding the ontological dimensions of methods, pointing towards methods use and performance as a
normatively-infused way of engaging with design practice.

5.1 Synthesis of Gaps in Current Knowledge of Ethics-Focused Methods

A synthesis of the results from the content analysis facilitates additional focus on complexities that exist in terms
of dissemination of the methods’ knowledge; codification of these methods with respect to design process; and the
performance of these methods in everyday practice. While our analytic approach does not allow us to resolve questions
or issues relating to dissemination challenges, we are able to observe a disconnect between the published formats of
most of these method sources and their intended audience. According to our results on the published format, 44.4%
of the methods were published in the form of academic papers, and the majority of these methods were created for
design practitioners or educators. However, this audience is frequently unable to access materials published in formal
academic venues due to paywall restrictions. The lack of the access and awareness of these methods from the perspective
of practitioners likely results in reduced adoption of the methods, even without accounting for other translational
barriers observed by HCI researchers [41, 87]. While our analytic approach does not allow us to resolve questions about
dissemination challenges, our critique depends on the quantifiable results of comparing the method’s intended audience
and published formats, which shows that most methods are behind a paywall. Additionally, as shown in Table 4, it is
evident that the majority of methods we analyzed were research-focused (Phases 2 and 4), generatively-focused (Phase
3 and 4), or evaluation-focused (Phase 4). This finding demonstrates the potential for exploring or creating method for
19
 Chivukula, et al.

Phase 1 (project planning focused) and phase 5 (monitoring focused) work. Creating methods focused on Phase 1 and 5
also have the potential to be more impactful in creating a space for ethically-focused work, inscribing the problem
space with ethical concerns (Phase 1) and continuously evaluating the work in a social context (Phase 5). One of the
rare example of methods involved in Phase 1 of the process includes the Ethical Contract method [68], which enables a
discussion among all the stakeholders on the project to clearly discuss and divide their “ethical responsibilities” prior to
concept generation. This method shows potential in discussing, communicating, and formalizing ethical responsibilities
across multiple stakeholders in everyday practice. As we provide the above critique, we also highlight limitations in our
coding approach. Our analysis was solely dependent on the available method descriptions text, and we worked to avoid
high levels of inference. We also rely on the qualitative results as we provide the gaps in the existing methods that are
more inclined towards Phases 2, 3, and 4, proposing that method developers and publishers considering using Table 2
as a framework to build, refine, or standardize their manuals.
Our findings also point towards the intended performance of these methods in everyday practice, revealing underlying
beliefs about practitioners, practice, and available resources. Methods we evaluated include mechanisms to aid in:
addressing power dynamics and solving complexities due to organizational rules (e.g., Data Ethics Canvas [3]); bringing
a balance between stakeholder requirements and designer intentions (e.g., A Value-Sensitive Action-Reflection Model
[154]); facilitating realization of designers’ ethical responsibilities and extending application of these responsibilities
beyond instructional settings (e.g., Re-Shape [127]); guiding through self-provocation to evaluate the impacts of create
technology (e.g., The Tarot of Tech [16]); bridging knowledge for practitioners from different disciplines (e.g., Idea
Generation through Empathy method [88]); monitoring impacts of shipped products (e.g., Design Ethically-Monitoring
Checklist [162]); and providing ethical or critical knowledge, concepts, and vocabulary to be applied to support decision
making (e.g. values through HuValue [98] and gender-inclusivity through GenderMag [36]). Our synthesis also reveals
several inscribed assumptions regarding the performance of these methods. First, the knowledge of these methods
are presented using types of guidance that are intended to encourage certain patterns of performance. For instance,
heuristics are used in the White Hat UX patterns [54] method, with the underlying assumption that these heuristics
can be applied without the impedance of existing business forces or other forms of complexity beyond the designer
themself. Second, the methods reveal beliefs that designers already have the vocabulary and capacity to express their
social responsibility and have the capability to take responsibility for positive social impact (e.g., as required to list in
Ethical Disclaimer [69]). Third, the methods reveal beliefs that practitioners are able to evaluate their decisions based
on concepts such as utility, human rights, and justice, which often resist quantification. These qualities of the intended
performance of methods’ knowledge in practice brings to the foreground the resonance of these methods with the
constraints and complexity of practice settings, and the capacity and existing knowledge of the practitioners. Building
on Stolterman’s [135] concept of rationality resonance, which highlights the relationship between suggested vs. existing
practice, we can further question the barriers to adoption of these methods in practice settings, identifying spaces
where existing assumed knowledge of practitioners is incomplete; spaces where the agency and power of designers is
not available to the extent that methods might assume; and spaces where ethical complexity across multiple stakeholder
positions is unaccounted for [83]. While we cannot resolve this issue of performance in the context of this paper, we
do propose that future work could address the role of methods as an emergent “new rationality” that could promote
ethical practices, while also guarding against methods descriptions and codification that lacks resonance with the
ethical design complexity present in everyday work practices.

20
Surveying the Landscape of Ethics-Focused Design Methods

5.2 Defining Ethics-Focused Methods

Prior design theory literature defines methods as a source of design knowledge that enables or supports design activity,
acting as a toolset to support the designer’s judgment and action throughout their design work. Based on the sensitizing
concepts and core of these methods, we have identified a range of underlying assumptions regarding the use of
these methods to discover knowledge, identify new possibilities, and locate hidden assumptions. These attitudes or
stances towards engagement with knowledge range from uncovering ethical components of a design situation or
problem through in situ speculation (e.g., Speculative Enactments [53]); considering ethical evaluation through acts
of iterating and futuring (e.g., The Oracle for Transfeminist Technologies [14]); fostering innovation through unique
value propositions by uncovering neglected or negatively impacted user groups in a use scenario (e.g., The Ethics
and Inclusion Framework [19]); influencing designers’ thinking to design for sustainable and non-deceptive behavior
change (e.g., Design with Intent [102]); helping to foreground and map the underlying intentions and world-view of a
designer (e.g., Description [67]); and providing a means for designers to operationalize ethics in their design process.
The primary aim of the knowledge contained in these methods was ethical impact—influenced by ethical theories,
values, and frameworks. We chose to define the collection as containing “ethics-focused methods” to differentiate these
methods from conventional design methods, conceptual frames, or theoretical commitments. In this way, the resulting
collection of 63 ethics-focused methods include prescriptive forms that are actionable and potentially performative on
the part of designers. Thus, the function of the method revealed through this embedded knowledge allows designers to
convert ethics-focused discovery into design outcomes. We have identified several means by which this translation
might occur, including: converting a prescribed value into a concern by describing how it is present in the design
context (e.g., Moral Value Map [71]); maximizing the ethical valence of a design situation by considering many ethical
theories together (e.g., Normative Design Scheme [72]); bringing legal, moral and ethical policies and values together
(e.g., Moral and Legal IT Deck [18]); introducing user-centric concepts into different disciplines (e.g., Idea Generation
through Empathy method [88]); broadening the scope of human values such as privacy (e.g., Privacy Futures through
Design Workbooks [149]); quantifying ethical decisions for users to improve their decision making (e.g., The Ethical
Design Scorecards [55]); and introducing critical and feminist constructs such as gender to expand the horizons of
technology design (e.g., Gender Mag [36]). This embedded knowledge aids the designer in translating complex ethical
concepts into normatively-informed work practices, bridging the liminal space between awareness and action.

6 IMPLICATIONS AND FUTURE WORK

Our findings include the identification of various descriptors embedded in existing ethics-focused methods, and the
discussion reveals even more complexities and underlying assumptions about the practical use of these methods that
relate to their specification, dissemination, and performance. Building on this work, there is substantial potential for
future research through investigation of issues relating to method adaption, evolution, and the resonance of these
methods with everyday work practices. Work in this area may lead to the identification of productive areas for the
creation of future ethics-focused methods, while also pointing towards barriers to adoption of existing methods in
practitioner discourses and work practices.
While we have identified a large set of ethics-focused methods, we do not claim that our collection is decidedly
complete. Due to the wide range of languaging approaches and dissemination strategies used in the set of methods we
were able to identify, collect, and analyze, we anticipate that there are likely other existing and emergent sources that
could enhance our collection. Thus, the implications and contribution of this work does not rest on the collection being
21
 Chivukula, et al.

objectively “complete”; rather, the primary contribution resides in the ways we have analyzed and attached descriptors
to these methods, revealing opportunities for new methods, refinement of existing methods, and means of standardizing
language among methods to increase portability and adaptation in everyday design work.
Further investigation into the practical use and popular awareness of these methods may provide opportunities for
method developers to consider alternative dissemination strategies. Future work may productively focus on studying
the creation of these methods, revealing the considerations method developers take into account, and the ways in which
they constrain or include aspects of ethical design complexity into the methods they create.
Building on our findings, we are able to identify opportunities for the creation of new ethics-focused methods, and
additional practices that may result in more resonant forms of methods dissemination, design process implementation,
iteration and adaptation, and translational opportunities among design practitioners, educators, and researchers. First,
we underscore the need to disseminate and distribute methods to the intended audience in more accessible and
public formats, including the potential creation of channels to sharing material between researcher and practitioner
communities. Second, we describe a substantial gap in the provision of design methods that support phases 1 and 5,
including ethical engagement with the framing of a problem space by considering ethical responsibilities across all
stakeholders (Phase 1) and iteratively evaluating a product in a social context after launching in the market (Phase 5).
The creation of new methods to address these spaces, or the intentional adaptation of existing methods to support these
forms of inquiry could bring substantial value to design conversations in these areas of practice. Third, methods could
be further evaluated in terms of their fit and portability, encouraging increased iterative use of a range of ethics-focused
methods across a range of design activities. The input->mechanic->output patterns reveal rich opportunities for this
interplay among methods, yet the languaging and media of the methods we have evaluated show distinct differences in
approach that make this ad hoc assemblages of methods difficult or unlikely under the pressures of everyday practice.
Fourth, we identify potential opportunities to build prescriptive methods that rely upon theoretical commitments
(e.g., theoretical commitments listed in Table 1), activating those concepts through new ethics-focused methods to
support design practice. Finally, we propose the creation or articulation of value-focused design frameworks for a
combination of stakeholders to build ethical alignment and engage members of a multi-disciplinary team, bringing
resonance across multiple practice contexts. Successfully engaging with these implications and provocations for future
work may substantively impact the ethical awareness of design and technology practitioners, while also pointing
towards needs and gaps in design education practices, where a core set of methods is often learned.

7 CONCLUSION
In this paper, we present a content analysis of 63 ethics-focused methods intended for use in design and technology
practice. We map the current landscape of ethical support and tools by characterizing the collection of these methods
along multiple dimensions, including the ways in which they operationalize ethics, their intended primary audience and
context of use, the core of the methods, their interaction qualities, and the ways in which these methods are articulated,
formulated, and languaged. We provide these methods as an initial collection, alongside a set of descriptors that mark
the existing landscape of ethical support for researchers, practitioners, and educators. We propose a definition for
ethics-focused methods and identify means of making these methods more resonant with HCI and design practice,
articulating multiple areas of future work to support method development and design practice.

ACKNOWLEDGMENTS
This work is funded in part by the National Science Foundation under Grant No. 1909714.
22
Surveying the Landscape of Ethics-Focused Design Methods

REFERENCES
[1] [n.d.]. ACM Code of Ethics and Professional Conduct. https://www.acm.org/code-of-ethics. Accessed: 2020-8-14.
[2] [n.d.]. AI Fairness 360. https://aif360.mybluemix.net/. Accessed: 2020-5-11.
[3] [n.d.]. The Data Ethics Canvas – The ODI. https://theodi.org/article/data-ethics-canvas/. Accessed: 2020-5-12.
[4] [n.d.]. Data Patterns Catalogue - IF. https://catalogue.projectsbyif.com/. Accessed: 2020-5-11.
[5] [n.d.]. Design Heuristics Cards: Strategies to Inspire Ideas. https://www.designheuristics.com/. Accessed: 2020-6-28.
[6] [n.d.]. Design Kit. https://www.designkit.org/methods. Accessed: 2020-8-28.
[7] [n.d.]. Diverse Voices: A How-To Guide for Creating More Inclusive Tech Policy Documents | Tech Policy Lab. https://techpolicylab.uw.edu/news/
diverse-voices-guide/. Accessed: 2020-5-12.
[8] [n.d.]. Ethical OS. https://ethicalos.org. https://ethicalos.org Accessed: 2020-9-13.
[9] [n.d.]. GenderMag Teach. https://sites.google.com/site/gendermagteach/. Accessed: 2020-5-12.
[10] [n.d.]. Inclusive Design Toolkit. http://www.inclusivedesigntoolkit.com/GS_evaluate/evaluate.html. Accessed: 2020-5-12.
[11] [n.d.]. Making an Ethical Decision. https://www.scu.edu/ethics-app/. Accessed: 2020-5-12.
[12] [n.d.]. Metaphor Cards- Template.
[13] [n.d.]. Microsoft Design. https://www.microsoft.com/design/inclusive/. Accessed: 2021-1-16.
[14] [n.d.]. The Oracle for Transfeminist Technologies. https://www.transfeministech.codingrights.org/. Accessed: 2021-1-16.
[15] [n.d.]. Responsible AI. https://www.microsoft.com/en-us/ai/responsible-ai?activetab=pivot1:primaryr6. Accessed: 2021-1-16.
[16] [n.d.]. The Tarot Cards Of Tech. http://tarotcardsoftech.artefactgroup.com/. Accessed: 2021-1-26.
[17] 2018. The dark side of UX Design: Practitioner-identified examples of stakeholder values superseding user values. https://darkpatterns.uxp2.com/.
Accessed: 2020-5-31.
[18] 2018. The Moral-IT & Legal-IT Decks. https://lachlansresearch.com/the-moral-it-legal-it-decks/. Accessed: 2020-5-12.
[19] 2019. The Ethics and Inclusion Framework. https://patriciagestoso.com/ethics-and-inclusion-framework/. Accessed: 2020-5-12.
[20] 2020. Investigating Consequences with Our Ethics Assessment. https://spotify.design/article/investigating-consequences-with-our-ethics-
assessment. Accessed: 2021-1-16.
[21] P Agre. 1997. Toward a critical technical practice: Lessons learned in trying to reform AI in Bowker. G. , Star, S. , Turner, W. , and Gasser, L. , eds,
Social Science, Technical Systems and Cooperative Work: Beyond the Great Divide, Erlbaum (1997).
[22] Taghreed Alshehri, Reuben Kirkham, and Patrick Olivier. 2020. Scenario Co-Creation Cards: A Culturally Sensitive Tool for Eliciting Values.
In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing
Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376608
[23] Aristotle. 2000. Aristotle: Nicomachean Ethics. Cambridge University Press.
[24] Stephanie Ballard, Karen M Chappell, and Kristen Kennedy. 2019. Judgment Call the Game: Using Value Sensitive Design and Design Fiction to
Surface Ethical Concerns Related to Technology. In Proceedings of the 2019 on Designing Interactive Systems Conference (San Diego, CA, USA) (DIS
’19). ACM, New York, NY, USA, 421–433. https://doi.org/10.1145/3322276.3323697
[25] Jeffrey Bardzell and Shaowen Bardzell. 2013. What is critical about critical design?. In Proceedings of the SIGCHI conference on human factors in
computing systems. 3297–3306.
[26] Jeffrey Bardzell, Shaowen Bardzell, and Mark A Blythe. 2018. Critical Theory and Interaction Design. MIT Press.
[27] Shaowen Bardzell. 2010. Feminist HCI: Taking Stock and Outlining an Agenda for Design. In Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’10). ACM, New York, NY, USA, 1301–1310. https://doi.org/10.1145/1753326.1753521
[28] Lawrence C Becker and Charlotte B Becker. 2001. Encyclopedia of ethics (2nd ed. ed.). Routledge, New York [N.Y.].
[29] Elettra Bietti. 2020. From ethics washing to ethics bashing: a view on tech ethics from within moral philosophy. In Proceedings of the 2020 Conference
on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT* ’20). Association for Computing Machinery, New York, NY, USA, 210–219.
https://doi.org/10.1145/3351095.3372860
[30] Michael Mose Biskjaer, Peter Dalsgaard, and Kim Halskov. 2010. Creativity methods in interaction design. In Proceedings of the 1st DESIRE Network
Conference on Creativity and Innovation in Design (Aarhus, Denmark) (DESIRE ’10). Desire Network, Lancaster, GBR, 12–21.
[31] Erling Björgvinsson, Pelle Ehn, and Per-Anders Hillgren. 2012. Agonistic participatory design: working with marginalised social movements.
CoDesign 8, 2-3 (June 2012), 127–144. https://doi.org/10.1080/15710882.2012.672577
[32] Mark Blythe. 2014. Research through design fiction: narrative in real and imaginary abstracts. In Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 703–712.
https://doi.org/10.1145/2556288.2557098
[33] Alan Borning, Batya Friedman, Janet Davis, and Peyina Lin. 2005. Informing Public Deliberation: Value Sensitive Design of Indicators for a
Large-Scale Urban Simulation. In ECSCW 2005. Springer Netherlands, 449–468. https://doi.org/10.1007/1-4020-4023-7_23
[34] Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (Jan. 2006), 77–101.
https://doi.org/10.1191/1478088706qp063oa
[35] Harry Brignull, Marc Miquel, Jeremy Rosenberg, and James Offer. 2015. Dark Patterns-User Interfaces Designed to Trick People.

23
 Chivukula, et al.

[36] Margaret Burnett, Simone Stumpf, Jamie Macbeth, Stephann Makri, Laura Beckwith, Irwin Kwan, Anicia Peters, and William Jernigan. 2016.
GenderMag: A Method for Evaluating Software’s Gender Inclusiveness. Interact. Comput. 28, 6 (Nov. 2016), 760–787. https://doi.org/10.1093/iwc/
iwv046
[37] Peter Buwert and Edinburgh Napier University. 2018. Examining the Professional Codes of Design Organisations. https://doi.org/10.21606/drs.
2018.493
[38] K Charmaz. 2008. Constructionism and the grounded theory method. Handbook of constructionist research (2008).
[39] Shruthi Sai Chivukula, Colin M Gray, and Jason A Brier. 2019. Analyzing Value Discovery in Design Decisions Through Ethicography. In Proceedings
of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). ACM, New York, NY, USA, 77:1–77:12.
https://doi.org/10.1145/3290605.3300307
[40] Shruthi Sai Chivukula, Chris Rhys Watkins, Rhea Manocha, Jingle Chen, and Colin M Gray. 2020. Dimensions of UX Practice that Shape Ethical
Awareness. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for
Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376459
[41] Lucas Colusso, Ridley Jones, Sean Munson, and Gary Hsieh. 2019. A Translational Science Model for HCI. In Proceedings of the 2019 CHI Conference
on Human Factors in Computing Systems (CHI ’19). ACM Press, New York, NY. https://doi.org/10.1145/3290605.3300231
[42] Sasha Costanza-Chock. 2020. Design Justice: Community-Led Practices to Build the Worlds We Need. MIT Press.
[43] Nigel Cross. 1984. Developments in design methodology. John Wiley & Sons.
[44] Robert Curedale. 2012. Design methods 1: 200 ways to apply design thinking. Design Community College Incorporated.
[45] Alexei Czeskis, Ivayla Dermendjieva, Hussein Yapit, Alan Borning, Batya Friedman, Brian Gill, and Tadayoshi Kohno. 2010. Parenting from
the pocket: value tensions and technical directions for secure and private parent-teen mobile safety. In Proceedings of the Sixth Symposium
on Usable Privacy and Security (Redmond, Washington, USA) (SOUPS ’10). Association for Computing Machinery, New York, NY, USA, 1–15.
https://doi.org/10.1145/1837110.1837130
[46] Vanessa Paz Dennen. 2004. Cognitive apprenticeship in educational practice: Research on scaffolding, modeling, mentoring, and coaching as
instructional strategies. Handbook of research on educational communications and technology 2, 2004 (2004), 813–828. http://faculty.weber.edu/
eamsel/Classes/Projects%20and%20Research%20(4800)/Teaching%20and%20Learning/Dennen%20(2004).pdf
[47] Tamara Denning, Batya Friedman, , and Tadayoshi Kohno. 2013. The Security Cards: A Security Threat Brainstorming Kit. http://securitycards.cs.
washington.edu/. Accessed: 2020-5-12.
[48] Michael Dieter and David Gauthier. 2019. On the Politics of Chrono-Design: Capture, Time and the Interface. Theory, Culture & Society 36, 2
(March 2019), 61–87. https://doi.org/10.1177/0263276418819053
[49] Catherine D’Ignazio and Lauren F Klein. 2020. Data Feminism. MIT Press.
[50] Clive Dilnot. 2005. Ethics? design? Archeworks.
[51] Carl DiSalvo. 2015. Adversarial Design. MIT Press.
[52] Anthony Dunne and Fiona Raby. 2013. Speculative Everything: Design, Fiction, and Social Dreaming. MIT Press.
[53] Chris Elsden, David Chatting, Abigail C Durrant, Andrew Garbett, Bettina Nissen, John Vines, and David S Kirk. 2017. On Speculative Enactments.
In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing
Machinery, New York, NY, USA, 5386–5399. https://doi.org/10.1145/3025453.3025503
[54] Trine Falbe, Kim Andersen, and Martin Michael Frederiksen. 2017. White Hat UX. Smashing Media AG.
[55] Trine Falbe, Martin Michael Frederiksen, and Kim Andersen. [n.d.]. The Ethical Design Scorecards. https://ethicaldesignhandbook.com/. Accessed:
2020-5-14.
[56] Mary Flanagan and Helen Nissenbaum. 2014. Values at Play in Digital Games. MIT Press, Cambridge, MA. https://market.android.com/details?id=
book-iIYRBAAAQBAJ
[57] Mary Flanagan and Helen Nissenbaum. 2014. Values at Play in Digital Games. MIT Press. https://doi.org/10.1177/1461444816631742
[58] Christopher Frauenberger, Marjo Rauhala, and Geraldine Fitzpatrick. 2017. In-Action Ethics. Interact. Comput. 29, 2 (March 2017), 220–236.
https://doi.org/10.1093/iwc/iww024
[59] Batya Friedman, Edward Felten, and Lynette I Millett. 2000. Informed consent online: A conceptual model and design principles. University of
Washington Computer Science & Engineering Technical Report 00–12–2 8 (2000).
[60] Batya Friedman and David Hendry. 2012. The envisioning cards: a toolkit for catalyzing humanistic and technical imaginations. In Proceedings of
the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI ’12). Association for Computing Machinery, New York,
NY, USA, 1145–1148. https://doi.org/10.1145/2207676.2208562
[61] Batya Friedman and David G Hendry. 2019. Value Sensitive Design: Shaping Technology with Moral Imagination. MIT Press.
[62] Batya Friedman, Peter Kahn, Jennifer Hagman, Rachel Severson, and Brian Gill. 2006. The Watcher and the Watched: Social Judgments About
Privacy in a Public Place. , 235–272 pages. https://doi.org/10.1207/s15327051hci2102_3
[63] Batya Friedman, Peter H Kahn, and Alan Borning. 2008. Value sensitive design and information systems. The handbook of information and computer
ethics (2008), 69–101.
[64] Batya Friedman and Peter H Kahn, Jr. 2003. Human values, ethics, and design. The human-computer interaction handbook (2003), 1177–1201.
[65] Batya Friedman, Ian Smith, Peter H. Kahn, Sunny Consolvo, and Jaina Selawski. 2006. Development of a Privacy Addendum for Open Source
Licenses: Value Sensitive Design in Industry. In UbiComp 2006: Ubiquitous Computing. Springer Berlin Heidelberg, 194–211. https://doi.org/10.
24
Surveying the Landscape of Ethics-Focused Design Methods

1007/11853565_12
[66] Bernard Gert. 1984. MORAL THEORY AND APPLIED ETHICS. Monist 67, 4 (1984), 532–548.
[67] Jet Gispen. 2017. De-scription. https://www.ethicsfordesigners.com/description. Accessed: 2020-5-12.
[68] Jet Gispen. 2017. Ethical Contract. https://www.ethicsfordesigners.com/ethical-contract. Accessed: 2020-5-12.
[69] Jet Gispen. 2017. Ethical Disclaimer. https://www.ethicsfordesigners.com/ethical-disclaimer. Accessed: 2020-5-12.
[70] Jet Gispen. 2017. Moral Agent. https://www.ethicsfordesigners.com/moral-agent. Accessed: 2020-5-12.
[71] Jet Gispen. 2017. Moral Value Map. https://www.ethicsfordesigners.com/moral-value-map. Accessed: 2020-5-12.
[72] Jet Gispen. 2017. Normative Design Scheme. https://www.ethicsfordesigners.com/normative-design-scheme. Accessed: 2020-5-12.
[73] Michelle Goddard. 2017. The EU General Data Protection Regulation (GDPR): European Regulation that has a Global Impact. International Journal
of Market Research 59, 6 (Nov. 2017), 703–705. https://doi.org/10.2501/IJMR-2017-050
[74] Gabriela Goldschmidt. 2014. Linkography: Unfolding the Design Process. MIT Press.
[75] Elizabeth Goodman. 2013. Delivering Design: Performance and Materiality in Professional Interaction Design. Ph.D. Dissertation. UC Berkeley.
https://escholarship.org/uc/item/31w6q6x3
[76] Elizabeth Goodman, Erik Stolterman, and Ron Wakkary. 2011. Understanding Interaction Design Practices. In Proceedings of the SIGCHI Conference
on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI ’11). ACM, New York, NY, USA, 1061–1070. https://doi.org/10.1145/1978942.
1979100
[77] Don Gotterbarn. 1998. Reconstructing the ACM code of ethics and teaching computer ethics. ACM SIGCSE Bulletin 30, 4 (Dec. 1998), 9–11.
https://doi.org/10.1145/306286.306293
[78] D W Gotterbarn, Bo Brinkman, Catherine Flick, Michael S Kirkpatrick, Keith Miller, Kate Vazansky, and Marty J Wolf. 2018. ACM Code of Ethics
and Professional Conduct. (2018).
[79] Colin M Gray. 2016. “It’s More of a Mindset Than a Method”: UX Practitioners’ Conception of Design Methods. In Proceedings of the 2016
CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). ACM, New York, NY, USA, 4044–4055. https:
//doi.org/10.1145/2858036.2858410
[80] Colin M Gray. 2016. What is the Content of ”Design Thinking”? Design Heuristics as Conceptual Repertoire. International Journal of Engineering
Education 32, 3B (2016), 1349–1355. http://www.ijee.ie/latestissues/Vol32-3B/05_ijee3220ns.pdf
[81] Colin M Gray. 2016. What is the nature and intended use of design methods. Proceedings of the Design Research Society (2016), 27–30.
[82] Colin M Gray and Elizabeth Boling. 2016. Inscribing ethics and values in designs for learning: a problematic. Educ. Technol. Res. Dev. 64, 5 (Oct.
2016), 969–1001. https://doi.org/10.1007/s11423-016-9478-x
[83] Colin M Gray and Shruthi Sai Chivukula. 2019. Ethical Mediation in UX Practice. In Proceedings of the 2019 CHI Conference on Human Factors in
Computing Systems (Glasgow, Scotland Uk) (CHI ’19). ACM, New York, NY, USA, 178:1–178:11. https://doi.org/10.1145/3290605.3300408
[84] Colin M Gray, Shruthi Sai Chivukula, and Ahreum Lee. 2020. What Kind of Work Do “Asshole Designers” Create? Describing Properties of Ethical
Concern on Reddit. In Proceedings of the 2020 ACM Designing Interactive Systems Conference (Eindhoven, Netherlands) (DIS ’20). Association for
Computing Machinery, New York, NY, USA, 61–73. https://doi.org/10.1145/3357236.3395486
[85] Colin M Gray, Yubo Kou, Bryan Battles, Joseph Hoggatt, and Austin L Toombs. 2018. The Dark (Patterns) Side of UX Design. In Proceedings of the
2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York,
NY, USA, 1–14. https://doi.org/10.1145/3173574.3174108
[86] Colin M Gray, Yubo Kou, Bryan Battles, Joseph Hoggatt, and Austin L Toombs. 2018. The Dark (Patterns) Side of UX Design. In Proceedings of the
2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York,
NY, USA, 1–14. https://doi.org/10.1145/3173574.3174108
[87] Colin M Gray, Erik Stolterman, and Martin A Siegel. 2014. Reprioritizing the relationship between HCI research and practice: bubble-up and
trickle-down effects. In Proceedings of the 2014 Conference on Designing Interactive Systems (Vancouver, BC, Canada) (DIS ’14). ACM, New York,
New York, USA, 725–734. https://doi.org/10.1145/2598510.2598595
[88] Colin M Gray, Seda Yilmaz, Shanna R Daly, Colleen M Seifert, and Richard Gonzalez. 2015. Idea Generation Through Empathy: Reimagining the
‘Cognitive Walkthrough’. (2015).
[89] Bruce Hanington. 2003. Methods in the Making: A Perspective on the State of Human Research in Design. Design Issues 19, 4 (Oct. 2003), 9–18.
https://doi.org/10.1162/074793603322545019
[90] Bruce Hanington and Bella Martin. 2012. Universal Methods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design
Effective Solutions. Rockport Publishers.
[91] S Harrison, M Back, and D Tatar. 2006. It’s Just a Method!: a pedagogical experiment in interdisciplinary design. In Proceedings of the 6th conference
on Designing Interactive systems. ACM, 261–270. https://doi.org/10.1145/1142405.1142445
[92] Justin L Hess and Grant Fore. 2018. A Systematic Literature Review of US Engineering Ethics Interventions. Science and engineering ethics 24, 2
(April 2018), 551–583. https://doi.org/10.1007/s11948-017-9910-6
[93] Hsiu-Fang Hsieh and Sarah E Shannon. 2005. Three approaches to qualitative content analysis. Qual. Health Res. 15, 9 (Nov. 2005), 1277–1288.
https://doi.org/10.1177/1049732305276687
[94] Lilly Irani, Janet Vertesi, Paul Dourish, Kavita Philip, and Rebecca E Grinter. 2010. Postcolonial computing: a lens on design and development.
In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA,
25
 Chivukula, et al.

1311–1320. https://doi.org/10.1145/1753326.1753522
[95] John Chris Jones. 1992. Design Methods. John Wiley & Sons.
[96] Immanuel Kant. 1785. Groundwork of the Metaphysics of Morals.
[97] Os Keyes, Jevan Hutson, and Meredith Durbin. 2019. A Mulching Proposal: Analysing and Improving an Algorithmic System for Turning the
Elderly into High-Nutrient Slurry. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk)
(CHI EA ’19). ACM, New York, NY, USA, alt06:1–alt06:11. https://doi.org/10.1145/3290607.3310433
[98] Shadi Kheirandish, Mathias Funk, Stephan Wensveen, Maarten Verkerk, and Matthias Rauterberg. 2019. HuValue: a tool to support design students
in considering human values in their design. Int. J. Technol. Des. Educ. (May 2019). https://doi.org/10.1007/s10798-019-09527-3
[99] S Kolarić, J Beck, and E Stolterman. 2020. On the Hierarchical Levels of Design Knowledge. Proceedings of the Design Society: DESIGN Conference 1
(May 2020), 51–60. https://doi.org/10.1017/dsd.2020.330
[100] John C Lere and Bruce R Gaumnitz. 2003. The Impact of Codes of Ethics on Decision Making: Some Insights from Information Economics. J. Bus.
Ethics 48, 4 (Dec. 2003), 365–379. https://doi.org/10.1023/B:BUSI.0000005747.37500.c8
[101] A Light. 2011. HCI as heterodoxy: Technologies of identity and the queering of interaction with computers. Interact. Comput. 23, 5 (Sept. 2011),
430–438. https://doi.org/10.1016/j.intcom.2011.02.002
[102] Dan Lockton. 2010. Design with Intent: 101 Patterns for Influencing Behaviour Through Design. Equifine.
[103] Nick Logler, Daisy Yoo, and Batya Friedman. 2018. Metaphor Cards: A How-to-Guide for Making and Using a Generative Metaphorical Design
Toolkit. In Proceedings of the 2018 Designing Interactive Systems Conference (Hong Kong, China) (DIS ’18). Association for Computing Machinery,
New York, NY, USA, 1373–1386. https://doi.org/10.1145/3196709.3196811
[104] Jonas Löwgren. 2013. Annotated portfolios and other forms of intermediate-level knowledge. Interactions 20, 1 (Jan. 2013), 30–34. https:
//doi.org/10.1145/2405716.2405725
[105] Jonas Löwgren and Erik Stolterman. 1999. Methods & Tools: Design Methodology and Design Practice. 6, 1 (Jan. 1999), 13–20. https://doi.org/10.
1145/291224.291233
[106] Jamie Luguri and Lior Strahilevitz. 2019. Shining a Light on Dark Patterns. (Aug. 2019). https://doi.org/10.2139/ssrn.3431205
[107] Noëmi Manders-Huits and Michael Zimmer. 2009. Values and pragmatic action: The challenges of introducing ethical intelligence in technical
design communities. International Review of Information Ethics 10, 2 (2009), 37–45.
[108] Arunesh Mathur, Gunes Acar, Michael J Friedman, Elena Lucherini, Jonathan Mayer, Marshini Chetty, and Arvind Narayanan. 2019. Dark
Patterns at Scale: Findings from a Crawl of 11K Shopping Websites. Proc. ACM Hum.-Comput. Interact. 3, CSCW (Nov. 2019), 1–32. https:
//doi.org/10.1145/3359183
[109] Joshua Mauldin. 2018. Black Mirror brainstorms — a product design exercise. https://uxdesign.cc/black-mirror-brainstorms-f919ccf5938c. Accessed:
2020-5-14.
[110] Nick Merrill. 2020. Security Fictions: Bridging Speculative Design and Computer Security. In Proceedings of the 2020 ACM Designing Interactive
Systems Conference (Eindhoven, Netherlands) (DIS ’20). Association for Computing Machinery, New York, NY, USA, 1727–1735. https://doi.org/10.
1145/3357236.3395451
[111] Nick Merrill and Joanne Ma. [n.d.]. Adversary Personas. https://daylight.berkeley.edu/adversary-personas/. Accessed: 2020-5-14.
[112] Jessica K Miller, Batya Friedman, Gavin Jancke, and Brian Gill. 2007. Value tensions in design: the value sensitive design, development, and
appropriation of a corporation’s groupware system. In Proceedings of the 2007 international ACM conference on Supporting group work (Sanibel
Island, Florida, USA) (GROUP ’07). Association for Computing Machinery, New York, NY, USA, 281–290. https://doi.org/10.1145/1316624.1316668
[113] Michael Mose Biskjaer, Peter Dalsgaard, and Kim Halskov. 2017. Understanding creativity methods in design. In Proceedings of the 2017 conference
on designing interactive systems. 839–851.
[114] Maurice Mulvenna, Jennifer Boger, and Raymond Bond. 2017. Ethical by Design: A Manifesto. In Proceedings of the European Conference on
Cognitive Ergonomics 2017 (Umeå, Sweden) (ECCE 2017). ACM, New York, NY, USA, 51–54. https://doi.org/10.1145/3121283.3121300
[115] Sean A Munson, Daniel Avrahami, Sunny Consolvo, James Fogarty, Batya Friedman, and Ian Smith. 2011. Attitudes toward online availability
of US public records. In Proceedings of the 12th Annual International Digital Government Research Conference: Digital Government Innovation in
Challenging Times (College Park, Maryland, USA) (dg.o ’11). Association for Computing Machinery, New York, NY, USA, 2–9. https://doi.org/10.
1145/2037556.2037558
[116] Lisa P Nathan. 2012. Sustainable information practice: An ethnographic investigation. J. Am. Soc. Inf. Sci. Technol. 63, 11 (Nov. 2012), 2254–2268.
https://doi.org/10.1002/asi.22726
[117] Lisa P Nathan, Predrag V Klasnja, and Batya Friedman. 2007. Value scenarios: a technique for envisioning systemic effects of new technologies. In
CHI ’07 Extended Abstracts on Human Factors in Computing Systems (San Jose, CA, USA) (CHI EA ’07). Association for Computing Machinery, New
York, NY, USA, 2585–2590. https://doi.org/10.1145/1240866.1241046
[118] Kimberly A Neuendorf. 2016. The Content Analysis Guidebook. SAGE.
[119] Chris Nodder. 2013. Evil by Design: Interaction Design to Lead Us into Temptation. John Wiley & Sons, Indianapolis, IN. 303 pages.
[120] Stuart Reeves. 2019. How UX Practitioners Produce Findings in Usability Testing. ACM transactions on computer-human interaction: a publication
of the Association for Computing Machinery 26, 1 (2019), 1–38. https://doi.org/10.1145/3299096
[121] Wessel Reijers, Killian Levacher, and Arturo Calvo. [n.d.]. The Ethics Canvas. https://www.ethicscanvas.org/canvas/index.php. Accessed: 2020-5-12.
[122] Horst Rittel. 1984. Second-Generation Design Methods. Developments in Design Methodology (1984), 317–327.
26
Surveying the Landscape of Ethics-Focused Design Methods

[123] Johnny Saldana. 2015. The Coding Manual for Qualitative Researchers. SAGE. https://market.android.com/details?id=book-ZhxiCgAAQBAJ
[124] Elizabeth Sanders and Pieter Jan Stappers. 2012. Convivial Toolbox: Generative Research for the Front End of Design. BIS.
[125] Donald A Schön. 1990. The design process. In Varieties of thinking: Essays from Harvard’s philosophy of education research center, V A Howard (Ed.).
Routledge, 111–141.
[126] Phoebe Sengers, Kirsten Boehner, Shay David, and Joseph ’jofish’ Kaye. 2005. Reflective Design. In Proceedings of the 4th Decennial Conference on
Critical Computing: Between Sense and Sensibility (Aarhus, Denmark) (CC ’05). ACM, New York, NY, USA, 49–58. https://doi.org/10.1145/1094562.
1094569
[127] Ben Rydal Shapiro, Amanda Meng, Cody O’Donnell, Charlotte Lou, Edwin Zhao, Bianca Dankwa, and Andrew Hostetler. 2020. Re-Shape: A Method
to Teach Data Ethics for Data Science Education. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI,
USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376251
[128] Ben Rydal Shapiro, Amanda Meng, Edwin Zhao, Charlotte Lou, Cody O’Donnell, and & Bianca Dankwa. [n.d.]. Re-Shape. https://www.benrydal.
com/re-shape. Accessed: 2020-5-19.
[129] Katie Shilton. 2013. Values Levers: Building Ethics into Design. Sci. Technol. Human Values 38, 3 (May 2013), 374–397. https://doi.org/10.1177/
0162243912436985
[130] Katie Shilton. 2018. Values and Ethics in Human-Computer Interaction. Foundations and Trends® Human–Computer Interaction 12, 2 (2018),
107–171. https://doi.org/10.1561/1100000073
[131] Kristen Shinohara, Nayeri Jacobo, Wanda Pratt, and Jacob O Wobbrock. 2020. Design for Social Accessibility Method Cards: Engaging Users
and Reflecting on Social Scenarios for Accessible Design. ACM Trans. Access. Comput. Article 17 (Jan. 2020). https://doi.org/blication:
ACMTransactionsonAccessibleComputingJanuary2020ArticleNo.:17https://doi.org/10.1145/3369903
[132] Michael Stallings. [n.d.]. Make it (Critical). https://crit.luddy.indiana.edu/design/. Accessed: 2020-5-14.
[133] Luke Stark and Kate Crawford. 2019. The Work of Art in the Age of Artificial Intelligence: What Artists Can Teach Us About the Ethics of Data
Practice. 1 17, 3/4 (Sept. 2019), 442–455. https://doi.org/10.24908/ss.v17i3/4.10821
[134] Marc Steen. 2015. Upon Opening the Black Box and Finding It Full: Exploring the Ethics in Design Practices. Sci. Technol. Human Values 40, 3 (May
2015), 389–420. https://doi.org/10.1177/0162243914547645
[135] E Stolterman. 2008. The nature of design practice and implications for interaction design research. International Journal of Design 2, 1 (Jan. 2008),
55–65. https://doi.org/10.1016/j.phymed.2007.09.005
[136] Erik Stolterman, Jamie McAtee, David Royer, and Selvan Thandapani. 2008. Designerly tools. In Undisciplined! Design Research Society Conference
2008 (Sheffield, UK). Sheffield Hallam University, Sheffield, UK, 116:1–14.
[137] Erik Stolterman and James Pierce. 2012. Design Tools in Practice: Studying the Designer-Tool Relationship in Interaction Design. In Proceedings of
the Designing Interactive Systems Conference on - DIS ’12. ACM Press, New York, New York, USA, 25–28. https://doi.org/10.1145/2317956.2317961
[138] Jason Chew Kit Tham. 2019. Feminist design thinking: a norm-creative approach to communication design. In Proceedings of the 37th ACM
International Conference on the Design of Communication (Portland, Oregon) (SIGDOC ’19). Association for Computing Machinery, New York, NY,
USA, 1–3. https://doi.org/10.1145/3328020.3353919
[139] Martin Tomitsch, Cara Wrigley, Madeleine Borthwick, Naseem Ahmadpour, Jessica Frawley, A Baki Kocaballi, Claudia Nunez-Pacheco, and Karla
Straker. 2018. Design. Think. Make. Break. Repeat. A handbook of methods. BIS Publishers, The Netherlands.
[140] Annemiek Van Boeijen, Jaap Daalhuizen, Roos van der Schoor, and Jelle Zijlstra. 2014. Delft design guide: Design strategies and methods. BIS
Publishers.
[141] Aimee van Wynsberghe and Scott Robbins. 2014. Ethicist as designer: a pragmatic approach to ethics in the lab. Sci. Eng. Ethics 20, 4 (Dec. 2014),
947–961. https://doi.org/10.1007/s11948-013-9498-4
[142] Peter-Paul Verbeek. 2005. What Things Do: Philosophical Reflections on Technology, Agency, and Design. Penn State Press.
[143] Arnold POS Vermeeren, Effie Lai-Chong Law, Virpi Roto, Marianna Obrist, Jettie Hoonhout, and Kaisa Väänänen-Vainio-Mattila. 2010. User
experience evaluation methods: current state and development needs. In Proceedings of the 6th Nordic conference on human-computer interaction:
Extending boundaries. 521–530.
[144] Kari Edison Watkins, Alan Borning, G Scott Rutherford, Brian Ferris, and Brian Gill. 2013. Attitudes of bus operators towards real-time transit
information tools. , 961–980 pages. https://doi.org/10.1007/s11116-013-9450-0
[145] Anne-Marie Willis. 2006. Ontological Designing. Design Philosophy Papers 4, 2 (June 2006), 69–92. https://doi.org/10.2752/
144871306X13966268131514
[146] Jill Palzkill Woelfer, Amy Iverson, David G Hendry, Batya Friedman, and Brian T Gill. 2011. Improving the safety of homeless young people
with mobile phones: values, form and function. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC,
Canada) (CHI ’11). Association for Computing Machinery, New York, NY, USA, 1707–1716. https://doi.org/10.1145/1978942.1979191
[147] Marty J Wolf, Don Gotterbarn, and Michael S Kirkpatrick. 2019. ACM Code of Ethics: Looking Back and Forging Ahead. In Proceedings of
the 50th ACM Technical Symposium on Computer Science Education (Minneapolis, MN, USA) (SIGCSE ’19). ACM, New York, NY, USA, 801–802.
https://doi.org/10.1145/3287324.3287519
[148] Richmond Y Wong. 2021. Using Design Fiction Memos to Analyze UX Professionals’ Values Work Practices: A Case Study Bridging Ethnographic and
Design Futuring Methods. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411764.3445709

27
 Chivukula, et al.

[149] Richmond Y Wong, Deirdre K Mulligan, Ellen Van Wyk, James Pierce, and John Chuang. 2017. Eliciting Values Reflections by Engaging Privacy
Futures Using Design Workbooks. Proc. ACM Hum. -Comput. Interact. 1, CSCW (Dec. 2017), 111:1–111:26. https://doi.org/10.1145/3134746
[150] Richmond Y Wong and Tonya Nguyen. 2021. Timelines: A World-Building Activity for Values Advocacy. In Proceedings of the 2021 CHI Conference
on Human Factors in Computing Systems. https://doi.org/10.1145/3411764.3445447
[151] Daisy Yoo. 2017. Stakeholder Tokens: A Constructive Method for Value Sensitive Design Stakeholder Analysis. In Proceedings of the 2017 ACM
Conference Companion Publication on Designing Interactive Systems (Edinburgh, United Kingdom) (DIS ’17 Companion). Association for Computing
Machinery, New York, NY, USA, 280–284. https://doi.org/10.1145/3064857.3079161
[152] Daisy Yoo. 2018. Stakeholder Tokens: a constructive method for value sensitive design stakeholder analysis. Ethics Inf. Technol. (2018), 1–5.
[153] Daisy Yoo, Katie Derthick, Shaghayegh Ghassemian, Jean Hakizimana, Brian Gill, and Batya Friedman. 2016. Multi-lifespan Design Thinking: Two
Methods and a Case Study with the Rwandan Diaspora. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San
Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 4423–4434. https://doi.org/10.1145/2858036.2858366
[154] Daisy Yoo, Alina Huldtgren, Jill Palzkill Woelfer, David G Hendry, and Batya Friedman. 2013. A value sensitive action-reflection model: evolving a
co-design space with stakeholder and designer prompts. In Proceedings of the SIGCHI conference on human factors in computing systems. 419–428.
[155] Meg Young, Lassana Magassa, and Batya Friedman. 2019. Toward inclusive tech policy design: a method for underrepresented voices to strengthen
tech policy documents. Ethics Inf. Technol. 21, 2 (June 2019), 89–103. https://doi.org/10.1007/s10676-019-09497-z
[156] Kat Zhou. 2021. 360 Review. https://www.designethically.com/360-review. Accessed: 2021-1-16.
[157] Kat Zhou. 2021. BJ Fogg’s Inverted Behavior Model. https://www.designethically.com/bj-fogg. Accessed: 2021-1-16.
[158] Kat Zhou. 2021. Dichotomy Mapping. https://www.designethically.com/dichotomy-mapping. Accessed: 2021-1-16.
[159] Kat Zhou. 2021. Hippocratic Oath. https://www.designethically.com/hippocratic-oath. Accessed: 2021-1-16.
[160] Kat Zhou. 2021. Layers of Effect. https://www.designethically.com/layers. Accessed: 2021-1-16.
[161] Kat Zhou. 2021. Maslow Mirrored. https://www.designethically.com/maslow-mirrored. Accessed: 2021-1-16.
[162] Kat Zhou. 2021. Monitoring Checklist. https://www.designethically.com/monitoring. Accessed: 2021-1-16.
[163] Kat Zhou. 2021. Motivation Matrix. https://www.designethically.com/motivation-matrix. Accessed: 2021-1-16.

28