Model Fairness

Instructor: Saravanan Thirumuruganathan

 Need for Fairness in ML
• People build ML systems to automate decision making, so machines
can make decision instead of people.

• Ideal: Automated decisions can be cheaper, more accurate, more

impartial, improve our lives

• Reality: If we aren’t careful, automated decisions can encode bias,

harm people, make lives worse

Justin Johnson & David Fouhey March 25, 2021EECS 442 WI 2021
Harms of ML Systems
• Allocative Harms : a ML system that is used to allocate resource can
allocate it unfairly or perpetuate inequality
• Sentencing criminals, loan applications, insurance rates, job applications

• Representational Harms : The ML system can reinforce harmful

stereotypes
• Men are CEOs, women are nurses
 Allocative Harms
• COMPAS for Recidivism
• Propublica investigated the
bias COMPAS exhibited
against black people.
• Caveat: Story is a bit more
complex than this.

h/t: Sahil Verma . https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing

 Allocative Harms
• Amazon discovered that the
algorithm for rating
candidates for developer
jobs was biased against
women.

h/t: Sahil Verma . https://www.theguardian.com/technology/2018/oct/10/amazon-hiring-ai-gender-bias-recruiting-engine

 Allocative Harms
• Health care risk
management!
• The algorithm used to
allocate high risk health care
was found to be biased
against black people.
• It has been used for over
200 M patients.

h/t: Sahil Verma . https://www.scientificamerican.com/article/racial-bias-found-in-a-major-health-care-risk-algorithm/

 Allocative Harms
• Predictive Policing
• The algorithm used to
predict regions with high
probability of crime was
found to be biased against
black people.

h/t: Sahil Verma . https://www.theverge.com/2021/12/6/22814409/go-read-this-gizmodo-analysis-predpol-software-disproportionate-algorithm

 Allocative Harms
• Amazon Prime
• Amazon one day delivery
service was found to be
biased against black
neighborhoods across US.
• Caveat: Amazon used the
ZIP code to decide one day
delivery, not the
demographics.

h/t: Sahil Verma . https://www.bloomberg.com/graphics/2016-amazon-same-day/

 Allocative Harms
• Facebook algorithm for ads was
found to be biased against
women and minorities.
• Job ads for janitors and taxi
drivers shown to minorities.
• Ads for nurses and secretaries
shown more to women.
• Home sale jobs shown to white
users, while rentals shown to
minorities.

h/t: Sahil Verma . https://www.technologyreview.com/2019/04/05/1175/facebook-algorithm-discriminates-ai-bias/

 Representational Harms
• Google Photos mistakenly
tagged Black people as gorillas.
• Google fixed this problem by
banning words like “Gorilla”,
“Chimpanzee”, “monkeys” from
Google photos.

h/t: Sahil Verma . https://www.theverge.com/2018/1/12/16882408/google-racist-gorillas-photo-recognition-algorithm-ai

 Representational Harms

Source: https://www.reddit.com/r/europe/comments/m9uphb/hungarian_has_no_gendered_pronouns_so_google
 Representational Harms

Justin Johnson & David Fouhey March 25, 2021EECS 442 WI 2021
 Representational Harms

Justin Johnson & David Fouhey March 25, 2021EECS 442 WI 2021
 Representational Harms
Task: Gender Classification
Input: RGB Image
Output: {Man, Woman} Prediction

Buolamwini and Gebru, “Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification”, FAT* 2018
Fairness
• Intuition: are people are being treated equally?

• Defining “treated equally” is maddeningly hard

• There are 21 common definitions!
 Case Study: COMPAS
• Person commits a crime, is arrested
• COMPAS software predicts the chance that the person will commit
another crime in the future (recidivism)
• Recidivism scores are important: if a person is likely to commit
another crime, they can be denied a bail or get longer sentence
• Real system that has been used in New York, Wisconsin, California,
Florida, etc

Justin Johnson & David Fouhey March 25, 2021EECS 442 WI 2021
 Case Study: COMPAS
• 2016 ProPublica article analyzed COMPAS scores for >7000 people
arrested in Broward county, Florida
• Question: How many of these people ended up committing new
crimes within 2 years?

Justin Johnson & David Fouhey March 25, 2021EECS 442 WI 2021
 Error Metrics: Different Stakeholders

Justin Johnson & David Fouhey March 25, 2021EECS 442 WI 2021
 Case Study: COMPAS

Justin Johnson & David Fouhey March 25, 2021EECS 442 WI 2021
Case Study: COMPAS
• COMPAS gives very different outcomes for white vs black defendants,
but it does not use race as an input to the algorithm!

• Even if a sensitive feature (e.g. race) is not an input to the algorithm,

other features (e.g. zip code) may correlate with the sensitive feature
 Evaluating for Fairness
• Admission classification model: Majority (blue 80 students), Minority
(Orange 20 students)
• Goal: model must admit qualified students in a manner that is fair to
the candidates in both demographic groups.

https://developers.google.com/machine-learning/crash-course/fairness/evaluating-for-bias
 Demographic Parity
• Idea: compare the admissions rate for the majority group and the
minority group
• If it is (approximately) equal, a student's chance of being admitted to
the university doesn't vary by demographic group.

• In our example, a model exhibits demographic parity if it selects 16

candidates from majority group and 4 from minority. Both groups
have 20% acceptance rate

https://developers.google.com/machine-learning/crash-course/fairness/demographic-parity
 Demographic Parity

https://developers.google.com/machine-learning/crash-course/fairness/demographic-parity
 Demographic Parity Pros and Cons
• Demographic parity is independent from the target variable, there is
no need to have access to its data to measure and correct bias. So it is
suitable for applications when the target is hard to qualify
(employment qualification, credit default, justice, etc.)

• Demographic parity does not take the distribution of predictions for

each demographic group (the number of students classified as
"qualified" vs. "unqualified") into account when evaluating how the
20 admissions slots should be allocated.

https://developers.google.com/machine-learning/crash-course/fairness/demographic-parity
 Demographic Parity Issues

https://developers.google.com/machine-learning/crash-course/fairness/demographic-parity
 Demographic Parity Issues

https://developers.google.com/machine-learning/crash-course/fairness/demographic-parity
Equality of Opportunity
• Idea: compare the acceptance rates for just the qualified candidates
in the majority group and the minority group. If the acceptance rates
for qualified students in both groups are equal, the model exhibits
equality of opportunity

• Students with our preferred label ("qualified for admission") have an

equal chance of being admitted, irrespective of which demographic
group they belong to.
 Equality of Opportunity

Both groups have an acceptance rate of 40% for qualified students

Equality of Opportunity Issues
• Use where there is a clear-cut preferred label. If it's equally important
that the model predict both the positive class ("qualified for
admission") and negative class ("not qualified for admission") for all
demographic groups, it is not good (use equalized odds for this)

• Comparing error rates can only work when sensitive attribute

(majority/minority label) is available. This is not always possible.
• In US, asking for race/gender etc is illegal. So how can you measure fairness
without this info?
 Bias Taxonomy
• Reporting Bias occurs when the frequency of events, properties,
and/or outcomes captured in a dataset does not accurately reflect
their real-world frequency.

• Example: Suppose you want to train a sentiment classifier using

Goodreads reviews. The majority of reviews in the training dataset
reflect extreme opinions (reviewers who either loved or hated a
book), because people were less likely to submit a review of a book if
they did not respond to it strongly.

https://developers.google.com/machine-learning/crash-course/fairness/types-of-bias
 Bias Taxonomy
• Historical bias occurs when historical data reflects inequities that
existed in the world at that time.

• A city housing dataset from the 1960s contains home-price data that
reflects discriminatory lending practices in effect during that decade.

https://developers.google.com/machine-learning/crash-course/fairness/types-of-bias
 Bias Taxonomy
• Coverage bias occurs if data is not selected in a representative
fashion.

• Example: A model is trained to predict future sales of a new product

based on phone surveys conducted with a sample of consumers who
bought the product. Consumers who instead opted to buy a
competing product were not surveyed, and as a result, this group of
people was not represented in the training data.

https://developers.google.com/machine-learning/crash-course/fairness/types-of-bias
 Bias Taxonomy
• Non-response bias (also known as participation bias) occurs if data
ends up being unrepresentative due to participation gaps in the data-
collection process.

• A model is trained to predict future sales of a new product based on

phone surveys conducted with a sample of consumers who bought
the product and with a sample of consumers who bought a
competing product. Consumers who bought the competing product
were 80% more likely to refuse to complete the survey, and their data
was underrepresented in the sample.

https://developers.google.com/machine-learning/crash-course/fairness/types-of-bias
 Bias Taxonomy
• Sampling bias occurs if proper randomization is not used during data
collection.

• A model is trained to predict future sales of a new product based on

phone surveys conducted with a sample of consumers who bought
the product and with a sample of consumers who bought a
competing product. Instead of randomly targeting consumers, the
surveyor chose the first 200 consumers that responded to an email,
who might have been more enthusiastic about the product than
average purchasers.

https://developers.google.com/machine-learning/crash-course/fairness/types-of-bias
 Bias Taxonomy
• In-group bias is a preference for members of your own group you also
belong, or for characteristics that you also share.

• Two ML practitioners training a résumé-screening model for software

developers are predisposed to believe that applicants who attended
the same computer-science academy as they both did are more
qualified for the role.

https://developers.google.com/machine-learning/crash-course/fairness/types-of-bias
 Bias Taxonomy
• Out-group homogeneity bias is a tendency to stereotype individual
members of a group to which you do not belong, or to see their
characteristics as more uniform.

• Two ML practitioners training a résumé-screening model for software

developers are predisposed to believe that all applicants who did not
attend a computer-science academy don't have sufficient expertise
for the role.

https://developers.google.com/machine-learning/crash-course/fairness/types-of-bias
 Bias Taxonomy
• Implicit bias occurs when assumptions are made based on one's own
model of thinking and personal experiences that don't necessarily
apply more generally.

• An ML practitioner training a gesture-recognition model uses a head

shake as a feature to indicate a person is communicating the word
"no." However, in some regions of the world, a head shake actually
signifies "yes."

https://developers.google.com/machine-learning/crash-course/fairness/types-of-bias
 Bias Taxonomy
• Confirmation bias occurs when model builders unconsciously process
data in ways that affirm pre-existing beliefs and hypotheses.

• An ML practitioner is building a model that predicts aggressiveness in

dogs based on a variety of features (height, weight, breed,
environment). The practitioner had an unpleasant encounter with a
hyperactive toy poodle as a child, and ever since has associated the
breed with aggression. When curating the model's training data, the
practitioner unconsciously discarded features that provided evidence
of docility in smaller dogs.

https://developers.google.com/machine-learning/crash-course/fairness/types-of-bias
 Bias Taxonomy
• Experimenter's bias occurs when a model builder keeps training a model
until it produces a result that aligns with their original hypothesis.

• An ML practitioner is building a model that predicts aggressiveness in dogs

based on a variety of features (height, weight, breed, environment). The
practitioner had an unpleasant encounter with a hyperactive toy poodle as
a child, and ever since has associated the breed with aggression. When the
trained model predicted most toy poodles to be relatively docile, the
practitioner retrained the model several more times until it produced a
result showing smaller poodles to be more violent.

https://developers.google.com/machine-learning/crash-course/fairness/types-of-bias
Interlude
• Cybersecurity and VirusTotal
Mitigating Bias
• In many cases training dataset is biased, and collecting a less biased
dataset is not always possible.

• What can you do?

Pre-processing Techniques for Bias Mitigation
Approach: potentially unfair data are transformed into fair data, and a
standard classifier is applied

• Resampling rows of the data

• Reweighting rows of the data,
• Flipping the class labels across groups
• Omitting sensitive variables
• Omitting proxies
In-Processing Techniques for Bias Mitigation
• Afair model is learned directly from a potentially unfair dataset
• Modify the loss function to account for fairness constraints.
• A fair classifier must be developed for each distinct type of classifier

• Post processing
• Adjust the outputs of the model to abide by fairness criteria.
Post Processing Techniques for Bias Mitigation
• A standard classifier is first learned, and then the learned classifier is
modified to satisfy a fairness constraint
• Oblivious assumption: fair class labels are determined based only on
labels of a standard classifier and a sensitive value
 Fairness Tree

http://www.datasciencepublicpolicy.org/projects/aequitas/ and https://dssg.github.io/fairness_tutorial/Fairness_tutorial_slides_ic2s2_2021_version.pdf