Gender Bias

Report
 How can companies run tech
recruiting processes to support
a more diverse workplace?

We did a deep dive into our data to uncover However, we identified some factors to take
whether there are differences in performance into account that may ultimately improve the
between genders on Codility coding tests diversity of your technical hiring.
amongst different companies, and whether
Happy Hiring,
there are any factors that could systematically
help or hinder this.

We found that while there is a difference in

performance between genders, the data Aga Balcer-Thinlay
analyzed so far proves that there is not a gender Director of Product Research, Codility
bias in Codility tasks.

Gender Bias Report 02

 Contents About Codility
For 10 years Codility has been working with over 1.500+ businesses,

facilitating over 10 million skill assessment sessions on behalf of

our clients. Over the years, we’ve learned how to quickly assess
01 Score Distribution 04
How does performance differ between genders? programmer expertise and relative suitability for every open

position our clients are looking to fill. Codility created the tech

assessment movement and we’ve tested 3x more candidates than

02 Codility is unbiased 06
What elements of tasks impact performance? newer entrants in the space.

03 Candidate behavior 08
What behaviors are similar and different between genders?

04 Supporting diversity 11
How can companies improve diversity?

05 Key conclusions 13

Gender Bias Report 03

 01 distributio
Score
distribution

Gender
Gender Bias
Bias Report
Report
 Score Distribution
Score distribution

Is there a difference in We ran analysis on over 1 million candidate sessions and

performance between male found that yes, there is a difference in performance

and female candidates?

between genders.

Score Distribution for Male and Female Candidates Men Women Percentage of Female Candidates Invited to Codility Tests by Company
N=1,007,662 men + 154,468 women N=41 customers selected from 252 analyzed (274k sessions in 6486 tests)
58%
25% 60%

Performance Gap The customer with

Almost 5% more the highest
20%
female than male proportion of female
candidates score 10% 40% candidates invited
Proportion of Candidates (%)

or less, and overall, 5% to take a Codility

male candidates test was at 58%. The

Fraction of women (%)

have a 1-3% edge over customer with the
10%
female candidates in lowest proportion
20%
all scores over 50%. was at 2%.
12%
15%

2%
0% 0%
0-10% 10-20% 20-30% 30-40% 40-50% 50-60% 60-70% 70-80% 80-90% 90-100% Each bar represents a different company over a 12 months span. For the purposes of this graph,
we selected a range of companies across different market segments, sizes, and geolocations.
Codility Score (%)
What causes this difference?

Diversity is largely driven by the market. The biggest impact on the number of high-scoring female candidates occurs during The overall seniority gap between genders is also a contributing factor.
*See more in our
the initial sourcing phase, when the proportion of female candidates to male candidates is skewed. Approximately 81% of female developers have 6 years or less of 2018 Developer Report
professional coding experience,* and therefore are less likely to be
There are substantial differences in the proportion of different genders invited to take tests on the Codility platform between here
considered for senior developer roles.
customers.

Gender
Gender Bias
Bias Report
Report 0505
 02 unbiased
Codility
is unbiased

Gender
Gender Bias
Bias Report
Report
 Codility
Codilityis is
unbiased
unbiased

What elements of tasks impact We identified key task characteristics

which differentiate the way candidates
performance differently by gender? receive information.

Using these elements, we reviewed hundreds of Codility tasks to evaluate how these
Impact of Task Characteristics on a Candidate’s Score and Time Used
small variables affect gender.

Affect on score Affect on time used We found factors that had a slight correlation with candidate performance, but the

Man Woman Man Woman majority of factors impacted both genders in the same way.

The only factor that Math formulas

seemed skewed in favor of
females was the presence
of math formulas in the Pictures
task.
No other differences were
statistically relevant Complex data structures
enough to explain the
performance gap in a
meaningful way. Contains story

Difficult vocabulary

Level of difficulty

Legend

Length of description Very high High Neutral Low Very low

Gender
Gender Bias
Bias Report
Report 0707
 03 behavior
Candidate
behavior

Gender
Gender Bias
Bias Report
Report
 Candidade behavior
Candidate behavior

Both genders used time limits similarly,

How does usage of time began editing their code about 8 minutes
into the session, and typically started their
differ by gender? Codility test 3 days after the invitation
was sent.

We found that female candidates tend to give up on their session slightly more often
Candidate Start Time by Gender Men Women
than male candidates, with an 11% drop-off rate among females candidates vs. 9%
44.0% among males.
42.1%

40% Giving up is defined as using less than 20% of time and scoring less than 20%. The aver-

age rate of giving up across all candidates is 10%.

30%

22.6%
19.9%
20%
16.1%
Proportion of Candidates (%)

15.3%

12.3%
12.1%
8.5%
10%
7.1%

0%
Immediately 1st day 2nd day 3rd day After 3rd day
(first hour)
Time taken to start a session after invitation was received

Gender
Gender Bias
Bias Report
Report 0909
 Candidade behavior
Candidate behavior

How does code editing While both genders behave similarly when it comes to
time elements, there is a difference in how they build
differ by gender? and test their solution.

Number of Code Edits by Gender Men Women Score (%) by Number of Pastes into the Codility Interface
(score > 0%) N=365k male + 56k female

40% 60%
37.4%

Female candidates tend

33.2%
to edit their solution
more than male 40%
30%
candidates and run
tests more often.
23.9% This suggests female
22.6%
candidates 20%
double-check and fine
20%
tune their code more so

Score (%)
than male candidates.

0%
12.3%
11.4% 0 2 4 6 8 10
10.7%
9.7% Number of Pastes
10% Men Median score Women Median score
7.6%
6.5% Trendline for median score R=0.758 Trendline for median score R=0.702
5.2% 5.3%
3.4%
How does pasting code differ by gender?
Score (%)

1.0% 0.7% 1.7%

0% The more often candidates paste code, the lower the score they get, and this affects both genders similarly.
1-5 6-10 11-20 21-30 31-50 51-100 101-200 201-300 Pasting code often suggests a trial-and-error approach. On the other hand, pasting code just once, often
Number of Code Edits suggests the candidate has written, tested, and solved the task in their own editor and is ready to submit.

Gender
Gender Bias
Bias Report
Report 1010
 04 diversity
Supporting
diversity

Gender
Gender Bias
Bias Report
Report
 Supporting Diversity
Supporting diversity

How can companies The proportion of female candidates invited to a

Codility session has steadily increased over the past
improve diversity? 5 years.

Companies have the power to shape the diversity in their engineering teams
Proportion of Female Candidates Invited to Codility Over the Past 5 Years

01 You get what you put in. No matter how you run your tech recruitment processes, the biggest shift will
20% happen after you review existing sourcing strategies and develop new diversity-oriented tactics. You can
take a look at existing campaigns, events and job listings, review language and graphics to ensure
female candidates feel welcomed and excited to apply for your team.

15% 02 Ensure your tech recruiting team represents Do you help to recruit other developers
in your company?
the diverse workplace you desire. From the
sourcers to the interviewers to management, Men 50%
Yes
bringing diversity to your tech recruiting team Women 37%

will help encourage diversity in your pipeline. Codility 2018 Developers Report
10%

03 Review your existing tech recruiting tools, such as Codility, to confirm they are unbiased and support
your diversity efforts. Retrain key end-users on important tools to ensure they understand best-case
Proportion of women (%)

practices and are not unintentionally creating bias.

5%

04 Establish diversity as a core value of the company so hiring teams understand why it is significant and
how diversity brings value to your company.

0%
05 No matter how you are screening and interviewing your candidates, understand that male and female
1/1/2014 1/1/2015 1/1/2016 1/1/2017 1/1/2018 1/1/2019 candidates behave differently in screening tests. Once this is understood, your team can take these into
Month account when creating tests and reviewing candidates.
This is a trend driven by customers and is increasing at a faster rate than before.

Gender
Gender Bias
Bias Report
Report 1212
 05 conclusion
Key
conclusions

Gender
Gender Bias
Bias Report
Report
 Supporting Diversity
Key conclusions

Key We found that while there is a difference in performance

between genders, the data analyzed so far proves that
Conclusions there is no gender bias in Codility tasks.

There is a difference in The overall seniority gap There is a difference between The biggest impact on the Codility tasks were analyzed Over the past 5 years the
performance between between genders is an male and female behavior number of highly-scoring and do not create biased proportion of female
genders. aggravating factor. while building a solution in female candidates occurs outcomes between genders. candidates invited to Codility
Codility. during the initial sourcing sessions has steadily increased.
phase.

Almost 5% more female than Approximately 81% of female This difference suggests female There are substantial differences After analyzing over 1 million Continue to review and create
male candidates score 10% or developers have 6 years or less of candidates do more in the proportion of different sessions, no findings were tactics to improve diversity, like
lower, and overall, male professional coding experience, double-checking, fine-tuning, genders invited to the Codility statistically relevant enough to ensuring sourcing efforts
candidates have a 1-3% edge over therefore there is likely less and editing of their code after platform between customers - explain the performance gap in a represent your desired workplace
female candidates in all scores female participation in senior receiving feedback from running the lowest proportion is 2% meaningful way. demographics, and establish
over 50%. developer recruitment the solution. female candidates, and highest is diversity as a core company value.
campaigns. 58%.

Gender
Gender Bias
Bias Report
Report 1414
08