Technically Wrong Sexist Apps, Biased Algorithms, and Other Threats of Toxic Tech Sara Wachter-BoettcherChapter 1 Welcome to the Machine Openers tern ayant press, and you'll probably end up little excited, litle Dewildered and alot overwhelmed Yo sn downloadanapp to track your de, adjust your thermontat, and find a dog walker. Youcan order dinncr-andtipthe delivery person—without sa Inga word oF touching your wallet, You can subseribe to weekly rea ts, monthly st-seletod clothing deliveries, or even curated by none other than Bill Nye (yeah, the scence guy). The Ist goes on and on; whatever you And we are In late 2015, the Pew Research Cente reported say they're online “almost constant"! of them, My days usually start with bouncing between checking the news on Twitter and skimming my inbox for important st, While walking othe gym, I isten to the newsontheNPROneapp. Whilevaitingforthetraln, answer1 question from a teammate via Slack, the privatechatroom vervce. At some point, I'l pap over to Facebook to eatch up on tables and cats, nthe course ofa day. might make travel plans ‘heck ocal events lookup historical information get directions, and doa zion other things online Tim also part of the tech industry, In 2006, T was a twenty-three year-old aspiring writer from Oregon who had found here stuck inthe Arizona suburbs for a while, Thad ‘tumbled my way into ajob as copywriter at x small ad agen. ‘We marketed luxury realestate in planned, communities: .5,000-aquare-foot stucco homes with subzero appliances and Fafinity-edge pools. Within six months, Fa exhausted all he vrayvone could possibly describe x house ona golf course the (overt Then Taw the ad for web writer. wasn. sure what a teed writer di, to he honest. But I gure, why not? Ta been Tring the web sine AltaVista was the search engine of choice Besides, whatever the job was, it sounded hell of «lot more Interestingthan baking granite countertops Titus ut 2007 was a good ime to enter tech end uh. st as faraway fom realestate possible. Facebook wasjust start Ing to transform from allege-centic ste tothe behemoth its ince become. Fledgling messaging service twttr had just named itl Twitter, Google had just bought YouTbe, The {Phone ws bout to launch, Pretty soon wed be watching Vial ‘ideos, Fckrlling our friends, laughing at Joleats~and, of ‘ouree managingallthat mundane stuf like bankingand shop ping fom oa sereens And that meant pretty much every bus: eae wat aiming nt just toavea website but to igure out how technology might change the way it served it customers. ‘Sobere we are. edecadelatr.and technology Iso pervasive that aversion of peychologist Abraham Maslow’ hierarchy of news with“ WIP aded othe base of the pyramid has become ‘one ofthe mast enduring internet memes around ven tyre not someone who checks Instagram five times during dinner, you probably rely on connected technology forall ‘kinds ofthings you used to doinpersonor oer the phone. Appl {ng for job? Many companies onl take applications submited online. Contacting a business? How would you even find thelr phone number without visiting their site oF ski ing thee site oF asking Google? Booking tickets, researching a candidate, making a restaurant reservation fguringoutbus routes inane place—sure,youcan do all thse tasks ofine, but it's getting harder to manage it Gust try t ind good printed map of public rant in the next sityouvist)As technology shifted, 20 did my role~morphing fom rte something better than sick hero to “help us figure out why we have stl onthe we in the frst place, and how to com tnuneat there moreeflectivey."'veadvised startups, univets- ties, nonprofits, and even massive corporations that are stil strugglingto move tothe web. Ive designed large-scale websites, devised publishingstrategies, and used many more than my fai share of sticky notes and whiteboards to map out user flows o sete product features, ‘Eventually, thoughsomethingstartd to ee: Despite al ‘themprovements is technology my peers and I werent geting beter at serving peopl. Te relly hit me at the end of 201, when my fiend Ere Meyer—one of the web carly programmers and bloggers Toayed onto Racebook, It was Christmas Bve, and he expected the usual holiday photos and wel wishes from fiends and fami tie. Tastead, Facebook showed him an a fr its new Year In Review festre. ‘Year In Review allowed Facebook usersto create albums of thelr highlights fromthe year-top poss, photos rom vacations, ‘hat sort of thing-and share them wit their eiends, But Eric ‘wasnt ken on living 2014, the yearhis daughter Rebecca ded of aggressive rain canecr She wasskx. Facebook didn't give himachoee Instead treated sam ple Year In Review album for Nim, and posted it to his page to teacourage hit to share it, “Here's what your year looked ike! the copy read Below twas pietureofRebeces—the most opt tar photo Eve had posted ll yar. And surrounding Rebeceas iling face and curly hair were illustrations, made by Face book, of partersdaneingamid balloons and streamers, taecen who dedot crear hat yo (te Mee) He was gutted "Yes my yer looked ike that he wrote in Slate as is story went viral. "True enough. My yea loked like the n-absent face of my Lite Spark. twas stil unkind to remind meso tact leslyand without ny consent on my par Facebook had designed an experience that woeked wel fr people who'd hada goed year, people who had vacations or wed dings or parties to remember. But because the design team. focused onl on postive experiences, it adn’ thought enough about what would happen for everyone else—for people whose ‘years were marred by grief illness, heartbreak, ordinate; People lke Eriepald the pee1s not just Faeebook, and its not just grief or trauma, The more 1 started paying attention to how tech products are ‘designed, the more I started noticing how often they're full of blind spots, biases, and outright ethical Dunders—and how ‘often those oversights can exacerbate unfairnessand leeve ul erable people out Like inthe spring of 2015, when Louis ely. pediatrician sn Cambridge, England, joined PureGym, a British chain. But ‘everytime she red to swipe her membership card tosccess the ‘women'lockerroomshewasdenied-thesystemsimply wouldnt uthorie her. Finally Pury gto the bottom of things the third-party software it used to manage ts membership data software used tall ninety leationsacros England was rely Fn on member’ titles to determine which locker room they ‘could access And the tite "Doctor was coded st mal* (Orin Marchof 2016, when.JAMA eternal Medicine released study showing that the setiicial ntligence bul int smart ‘hones from Apple, Samsung. Google and Miroao init pro- frammedtobelpduringacrss, Thophones personal assistants fide understand words ike rape” oF "my husband shitting ‘me In fact, instead of dong even a simple web search, Sst “Apple's product-cracked jokes and mocked users” rast the first time Backin 201, fyoutol Sir youwere ‘inking about shooting yourself, twould give you directions ‘aun store. After geting bad press, Apple partnered with the National Sule Prevention Lifeline to offer users help when they sad something that iri ented suicidal But ive years tater no oe had looked beyond that one fx. Appiehad no prob Sie respnsee ow ei fqn March 2016 (Sam cherBowticha) lem investing in bulling jokes and clever comebacks into the interface from the star. But investing in crs or safety? Just nota priority. (Or in Angust 2016, when Snapehat Isunched & new fce-morphing flter-one it said was “insplted by anime.” In reality, the effect had slot more in common with Mickey Rooney playing I. ¥. Yunioshi in Breakfost at Tifan’ than a character from Akira. The ter morphed user’ selfs Into bcktoothed,squiny-eyedeareatures—the hallmarks of "yl Towtaee* the er for white people donning makeup and mas: qverading 98 Asian stereotypes. Sndpchat said that this articular filter wouldn't be coming back, but inssted it hadnt one anything wrong-even as Aslan users mounted a ean ign todelete the app.1 Tachniaty Wrong Msn ao ei of Mr Yt flee tf Gree Spares snownon he it er \Cension Groce Spuopen) Individually, Its eeey to write each of these off sa simple tipup-~a misstep. an oversight, shame. Weall make mistakes, right? Bt when we start ooking at them together, lear pat ten emerges: an industry tat is willing to invest plenty of. resources in chasing “delight” and “disruption” bu that hasnt stopped to think about who's being served by its products, and who's eine bend alienated or insulted. “There runningjoke in the HBO comedy Silicon Vly every ‘would-be entrepreneu, almost alwaysa twentysomething man. st some point announces that is product wil"make the world better place”—and then describes something elther absurdly ‘useless o technically trivial Cconstructing lesan her formaximumcode reuse and extenibiits"for example asurelt’ funny but [don actually watch the show regu lary I's too rea tbrings me back to too many treble conver tations at tech conferences some uy whos never eld ab in Bislifebacking me into coene and on shout hisdea to “disrupt” some industry or other, while 1 desperately sean the room foraway out ‘What Scr Valley etsrghtisthattechisan nsularindus try: a word of mostly white guys who've been tld they're specal-the best and brightest Isa story that tech ovesto tell bout ite and for god reason: the more everyone on the out doses technology asmagie and programmer as geniuses, the ‘more the industry ean keep doing whatover it wants, And with gobs of money and litle pbli seating, far to many people in tech have started to belive that theyre truly saving the wodd -Bven when they're jt making another ide-haling ap o es ‘aurant algorithm, Even when their products actually harm ‘more people than they hel. ‘Weeant afford that anymore. Te years ago tech was still, sn many ways, a discrete industry~easy to count and quantity. ‘Today, it’s more accurate to cal it core underpinning of very industry As tech entrepreneur and ativist Anil Dash writes, ‘very industry and every sector of society is powered by tech nology today, and being transformed by the choices made by technologist"* Including i'snow clear democracy Tim writing this in the wake of the 2016 presidential leetion~an election that gave usa Armes schies 3 cocktail hour and droningon10 Teeiclly wrong Infamous fr allegations of sexual asl, acim, eonflts of interest, collusion, and angry TWeetstorms, and wo rode to power anawave ofmisinformation. That misinformation was. at Teas in par, stoked by apolferation of fake-news stories and propaganda pieces that were picked up by social media algo- rithms and promoted as “trending” without any verifeation, ‘We can know fr sure how mich those stores, and ites like Facebook that put them In front of milions of readers, nu ‘enced the election. But too many of ws dont even know this is happening bs the frst place—beeause we simply don't know ‘enough about the design and technology choices hat shape out ‘world or the people who are making them, lfyourelike most people youprobably dont read Termsof Service agreements when you install new software out you ‘might ramble sbout how unintelligible they ae. You probably ‘ast know how Facebook decides which ads you should see (Cut youisht find itereepy when theystarttryingtosellyoua product youlooked at ast week. You probably don spend your ays deconstructing how your digital products were designed, and wi ‘But we can-and, a I'l show in this ook, we all must, Beene tech i omly going to become more fundamental to the ‘way we understand and intrse with our communities nd ov ‘ernments Courts are using software algorithms to influence rina sentencing Detailed medical records are being stored In databases, And, a information stules scholar Saya Noble puts it, "People are using search engines rather than ibrares or teachers to make sense ofthe world we're inhabiting” te not that ditrng the world is inherently ad, But the more technology becomes embedded in all aspect of life, the more it matters whether that technology isblased.slienting.or hasmful, The more it matter whether it works for eal people facing real-life stress The reat news is that understanding tech culture's ‘excesses, andthe effect they have on our digital lives is easier than yourigh think. Youdon't needa computer science degree ‘ora venture capital fund. You dont need tbe able to programm analgorithm,Allyouneed todo is slough away the ayers self: ‘ageandlzement and jargon, and get atthe heat of how people In technology work~and why their decisions so often don’t Inthis book we'l take acloser lookt how the tech indutey ‘operates, and se how its hiring practices and work culture re ste teams that don't represent most of s~no matter how many diversity” events these companies pt on “Then wel walkthrough way these design and development teams erate sallow perceptions of audiences and their needs, and how those perceptions lead to products tha, at best, leave fut huge pereentages of users~and at worst, take advantage of ‘ur personal data and encode bis int systems that hold tre ‘mendous poster over ea poopl's ives and livelihood. Alongthe way wellalso meet peopleandeompanies whore ‘uying to change things: The neighborhood based community site thats reducing racist posts hy changing the forms users ‘outon teste. The news organizationbuckingjournalemstrend toward shock and designing its app to prosde arty not anal ty. The emall- marketing platform thats focused on empathetic ‘communication rather than endless peppy pitches, By thetime we're done, Fhope youl se tech more ik do: ‘ota, but fallle—and pe for change. Bven more, Thopeyous comforablskinghard quent rs ‘Resjou se andthe people ho mae them, Beate as spat too ng making too many peopl el Hk here at is 5 mpurtastensghto design fr Bute wees there ing Chapter ‘wrong with you. There something wrong with tech Culture Misfit These centers ate meant to generate new product eas, exper rent with emerging technologies, and ultimately build proto types and products twas 2015. The first Apple Watch had just arvived. Smart ‘watch als were climbing And Fatima company wantedto get inon the action, It partnered with a major fashion brand, with the goal of designing women’s smartwatch-somethingfashion-forward toserveasanalterativetosllthe“ugly" wear ables that tech companies were launching. “As the project Keked of, Fatima sat down with the teams from both companies and was literally the only woman at he table Pretty soon, someone started a vdeo meant to show the prodets positioning twas al ash: yacht parties private et, $2,000 shoes. Fatima cringed, The smartwatch they were designing was meant to targot the midrange market—think Macy's ot Neiman Marcus “Lets walt until we get some research to make decisions she sid trying to pull he kckoTsssion back on track Noone paldattenton. Instead, shespentthe nexthourlistenngtooker rmentellherabout the “female market" usingtales oftheir wives shoppinghabltsaspro0t, ‘Bu Fatima dd give wp, Sho worked with colleague fom anothecofice to develop an aggressive research program, lea ininalghtsfrommorethana thousand peoplexerssthe United States and in couple other prominent markets. Fatima then brought their findings ck ote group. The team wanted to tanget women who are fashionable, tech-savvy. orboth. About two-thirds of respondents were den tifedasthe former andlfasthelater Except, themenrefuses to believe Fata, As soon as she stared presenting her data, ‘they wrote her off “Oh, Sl percent of the women ean be teh sav they sad “They werebescll ike, ‘Weare just goingto categorically ignore thethirtysomethingtechie because that pobably doesn't really exis" she said ater, “Even though, basedom ourreseareh, those were the people the mos key to buy thesmartwatch, Ccukure Miett15 nd (@).-.most likly to spend the largest amount of money on thosmarwateh" “The more se shared from the research dings, the more showas scoffed at. She deserbed how the women inthe research pools thatbeingabletodisreetly stay ontopofthings during ‘work meetings was criti. The mea in the room insisted that ‘mostwomen realyeareabout leisure-time activites. Sheshared that women reported rarely wsingshoppingappson theirphones, The men insisted thelr wives wete lays shopping, and the smartwatchabsolutely needed anapp for that Shereportedthat ‘the women sai functionality mattored most to thom-thatifit da’ work well nd full anced fashionable design wouldnt beenough The men nested, “Oh it doesn relly matter what tech weputinthere: "felt ike Twas in an episode of Mad Men” she told me (Over and over, he eas were discounted, and her expertise ignored. And as result, the sudience's actual needs-the ones identified and confirmed through her painstaking research were discarded, “That's specie project, a physical piece of technology, ‘that would exetin the world or not bared on whether hese men In the room accepted what I had to say or not” she sid, “They just werent willing to accept the research and use it as a foundation "The project got shelved and the brand partnered witha celebrity to designs smartwatch instead, It lopped ie wasn't based on needs; i was hised on stereotypes.” Fatima sad. “This was Tost opportunity for the people who could have used the smartwateh, but also for ths brand” Iwasalso lost opportunity for the innovation center: Pretty s00n, Fatima wastiredofhavingher ideas ignored She qu. Fatima story is over the top: her company ignored her Input made sexist assumptions, and lounched a product that fated Bat this mind-set—where someone assumes they have all leaves out anyone with adler theanswersaboutaproduc tent perspective—isnit rare, Scratch the surface a all Kinds of. ‘ompaniesfrom Silicon Valley's “uniomns” Gtartups with Yaluations of more tha aillion dollars) to tech im in lis round the workd—and you'll find a culture that routinely techs anyone who’ not young, white, and male, ‘One designer working on digital products in the Midwest told me she st down with her company’s owners to talk out maternity leave and found out they did even know whether they hada maternity policy Even though the company had forty ‘odd employees and had been business more than decade, no staff member had eve ben pregnant. In fact, only handful of women had ever worked thereat all. When she asked about cstablishing flex schedules and making work travel more pre: ‘ictal, she was shot down, “We have three other women of childbearing age on ou team, and we don't want tose a prece “dent” the omer told he, pregnancy were some sot of new trend. Sheba toqult~andsodidother women ho ot pregnant there afer shelf, So the company. which had sai it wanted to hire more women, stayed just as male-dominated as ever. "Another woman, from lane Bits teh company, tldme bout her firm's anmoal event in Las Vegas—part company retreat, part reraitment too for new hires. Before the bgp, marketing and finance, the two teams with lot of women on taf were sent anemallby aboard member asking them to“pat together some kind of dance rine to pesform a the company presenta” The woman iguored stant she otto the pre sentation. The heads of each department, all men, stood up and talked about thersuceesses over the course ofthe year. The only ‘women who graced the stage were a group of her pers in erop {opsand bot pants. The men nthe audience wolfhlstled while the women danced. When she complained, she was tol twas fine—no one had coerce them, ‘cam is rampant too, Take the story of product designer Amile Lamont, whose manager oneeelaimd she had seen herin a meeting “You're so black, you blend into the chat” she told? Or Brea Joy, aback sofware engineer who wrote that, past conorkers had constantly assume she was single mom. ‘ech i also known forts obsession with youthan obs iy ‘thirties male startup founders getting cosmetic surgery so that Investors will thnk they're sil n thei tents, Within these son so absurd that I now regularly heat rumors about e ‘companies this obsessionoftentakesthe form of group exer team runs, pushup contest, yo retreats. One man told me he found it 0 hard to keep up withthe younger men ot his team during his company’ forced workouts that he to quit. Other companies start their workdays, with alstaff mostngs held hile everyone does plaks—the fitness atvity where you get onthe ground, prop yourslfupby our feet andelbows,and old ‘he postion until your abs cant handle it anymore. I you're physically able to plank, that. And youre not wea nga dress Or feting modest. Or embarrased. Or vncomfortable geting on yourhands and knes at work, ‘Then, there'sthe alcohol. One woman told me she was pes sured to drinkso much t her welcome partythe Friday beforethr start datethat she spent the weekend efore her new job tren began recovering from mild sleohal poisoning, Another fold she hada coleague who startod bong al he impor tant meetings fr major project at the bar down the street. She doesn't drink, so he was never insted "You might think ha to work to get these stories, but no ‘When youre a woman workingin tec, they just come to yO, 8 evernding stream of frends and friends frends who just yrvetotellameone bout the atestidiulousshitthey encoun tered, And what all hese stories indicate tome is that, deste tech companies talking more and more about diversity fa too ‘much ofthe industry dosn't ulimatey care tha its practices dire making smart people feel uncomfortable, embarrassed, ‘unsafe or excluded. ‘With hescexampl sitivity ofso many teh products sxddenly make alot more sense. This isan industry that ean ook around at a bunch of young “white men who plank together in the mornings and get drunk Together inthe evenings and tink, This is great. This what & realty workplace looks like [echcalture does tntice howts alure excludes others=If cant even bother to listen to & ‘Noman a meeting why would notice when its prodvts do the same? Uatil he tech industry Becomes more representative ofthe peopeit'steyngto sere, these problems wil persist—and tourproduct willbe worseafbecase of sini, heel, sexism, andinse (CRAWLING TOWARD REPRESENTATION ‘Tobe fl, im not saying tech int doing anythin to improve verity, Youcan ind annual diversity reports from most ofthe Ccunure Mies 18 bisteompanies now highlightingshiftin employee demograph les and glossy profiles of staf from underrepresented groups. Whenever anow one comes out, though, it tends to read some thingie this one from Apple CEO Tim Cookin 2015 Wi oe proud ofthe progress we've made, and ritment to vest leurwaverng. But we know there ie & Orthiton om Fak fl detrei Maxie ‘Williams, in 2016: rth past ew yoo, we have boon working hard Yo 20 divest at Facebook tough avait of nr nal and tora programs and partnership, Wasi heve ‘long way to go" (r this one, from Naney Lee, Google's vice president of people operations in 2016 Wie sew encouraging sgn of progress in 2015, bt wete stil farfrom where we eed tobe ‘Theyall strike the sme tone: hopeful confident, maybe even — 15 Tim sure some PR rep somewhere Intended inspiring But ‘mean, thelr actual numbers? They barely shi. 12008, Apple leased its frst diversity eport-and made is commient fing he wok tat ns tl needs tobe done, At the time, was 70 sete done Ath 12870 perent male gobally, nd 80 percent male in technical roles, Two yas ater, in 2016,twastll 68 percent male globally and77 percent male in tech ea role? Similarly inthe United States, 9 pereent of Apple's ‘taf was black in 2016-though in leadership positions, that ‘number dropped to 8 percent~just the same as twas in 2014. Pls, te highest concentration of diverse employees won't be found at Apple shiny One Infinit Loop camps in California. “They work t your local mal, ingig up (Pads, explaining the ew MaeBooks, and checking you in for your Genius Bar ppolntment-not providing insight Into the design proces, or tvenbeng visible to those who are building products. ‘im not meaning to plc on Appl here in fut, they were actually one of the fist companies to release diversity data and thelr numbers look better than many others. For example, at Google, technical employees were 1 percent mal in 2016 ust percent were lack and percent were span. Inlesdership roles actos al departments, percent were male. Twopereent ‘ver black, and percent were Hispanie.OverstAirbnb, 1Oper Cont of staff came from “underrepresented groups” in 2016, (uhich means neither white nor Asin, the two groups that are ‘wellrepresentedintech companicd)butintechnical roles, that number was onlyS percent.” Tau o on but I don think you ned more stat soup understand this story. The numbers are mosty the same wher ‘rer you turn (cams tend tobe mach whiterand morema than the gencal population, and the skew s strongest in lesdership and technical positions. "Wight seem obvious why dlverse leadership matters hit ing women and people of color for only junior roles, and never promoting them, doesnt bode well for her ideas being valued tor theie perspectives having equal weight, But you might won- er, why do these companies stat always emphasize technical positions Grhich typically means people with ites Ike “eng nee “developer” or programmer”), whena wholehost of ath- rs are involved in eating a new digital product or service’? Here's why: in most tech companies, these rles~miich more than designers,copywriters, marketers, andothers who workon the creative or “of skills"end ofthe spectrum-are seenasthe rastermindsofnew technology: They'repaidthe best recruited ‘the hardest, and often have the most power on teams. While you're likely to find that staf bit more diverse outside of technical roles~and in particular, that women are better repre sentedin communications related jobs-those roles are histori cally undervalued (whichis @ whole other problem in tech culture but eave that or another da). PIPELINE DREAMS the tech industry has acknowledge thie problem and says it wants tof why ethestatssosiow to change? fyouask tech ie, they al point the same culprit theppetineThe term pipeline” refers to the numberof people who ae entering ‘thejob market prepared to join the tech industry those who sre learning to code in high school and gradusting fom computer ‘science or similar programs. Ifthe pipeline doesnt include enough women and people ooolor hough, honestly, many com: antes never get beyond talking about gender her) then tech ‘companies simply cant hire them. Ors the story ges, ‘Thats the argument Facebook used inthe summer of 2086, when i released yet anther report showing mlninl improve ‘mentsin diversity Gorexampl,only29percentofthenew seniorrire inthe year leading upto the report were women, anumber that barely changedthe eompsn's overall pctureofsenorlead teship, whichis just 27 percent womer).® “Appropriate FBT sentation n technology or ay other industry will depend upon nore people aving the opportunity to gin necessary skills through the public education syeten,”" sad Wiliams the diver sity head, who then wenn to expound on how few wornen and bck people study compute scence inhigh schoo! or cole. Kaya Thomas sees iferently. Backin October 204, when she was a sophomore computer science major at Dartmouth, she headed to Houston forthe Gance Hopper Celebration of Women in Computing—a massive ‘conference full famous speakers big budgets, and lot fc ference sa, She was ready. She had just completed an intern hip at Time Tne, where shed worked on a new spp for “Entertainment Weekly, She'd alo just unchedanPhone app of her own, We Read Too, which helps kids and teens nd books featuring people of coor She had worked Ina on-campus lb ‘uildng games. She had contributed to open-source code pro ets, And dhe putt allon résumé she hoped would catch the tention ofthe “ooltech companies that attend Grace Hopper torecruitineensand newstffersand get some good PRforsu~ porting women in tenologycompenies ike Twitter, Pinter ‘est, Apple, and Goole it eoemed perfect for Thomas. Butasshe walked aroundthe career firflor shedidn’tget the warm weleomeshel expected Infact, most ofthe recruiters didi even wantto seer résumé hey would avid looking erin the eye- Or tell ert 9 online and apply. Or turn aay total tosomeane els. And so she felt avis erased frm an eve that, she thought, was designe Ccltre mast 22 forpeoplelike her young women aiming tokickstartthelrtech- lca ances "Thomas had good reason to think Grace Hopper would lead to Internship opportunities, to, These companies talk end leesly about how hard it so fd enough programmes to fil thei positions. Other women toldherthey lef the event swim ring in job offers to choose from. But looking back, Thomas ‘alized that those women all ad something in common: they ‘werewhite Sheisblack Soshestatedtalkingwith other women of color and found that ther experiences were sii they fet Ignored oroverlookedin a ses f white faces es notjust Grace Hopper. You cant throw a pebble in Palo Ato without hitting a corporte-funded “versity” even these layslke the “Lean Tn irl that Facebook executive Sheryl Sandberysdvocatedinherbookotthesame name, othe wig tous code camps orkid from low-income homes put on by com: panies Ike Google. But what Thomas experienced convinced her that's not realy bout the pipeline, The tech industry just Isitlooking fr peopl ofealor—even when hose candidates are right infront of them, lke she was at Grace Hopper. Pu, mot tech recruiters gohackto thesame school, ver ‘and over-Stanford, Carnegie Mellon, MIT-rather than reaching out to places with more diverse student bodies (end strong computer slence departments). "Ifyou want to reruit ‘more new grads of color, send technealreruiters to Histor- cally Black Colleges and Universities and Hispanic Serving Institutes [ie” she wrote. *Stop blaming vs for not doing YOURjob."" ‘The numbers back her up. In 2014 analysis, USA Today concluded that “top universities turnout black and Hispanic1 Techsealy Wrong computer science and computer engineering graduates at twice the rate that leading technology companies ite them"* ‘Adding to the problems, Thomas say, potential employers spendthei meleokngfora“eulkureft"-someone who neatly hatches the employees already in the company-which ends up Teinforeing the status quo rater than changing it irick ood oetractrew grads, The fact that | dont the those things shouldnt mean im nota “cute ft | dov't want to workin tech foo round | want to create tmecing things and lar rom ther mart people. Thetis the cule fit you should bo ooking fo” ‘You might think she's overselling this concept of ulture ft” here but the perception iso widly shared, the phrase 9 COn- tantly used, hat t's even been spafed inthe Cooper Review's Honest Diversity in Tech Report” written by former Google ‘employee Sarah Cooper. “Our hing criteria ensures we havea liverse pool of canada” the post deadans. Then it shows pie chart where skils education and experience make upslivers ‘tthe hiring eritera, while “ability tot into the existing cu ture’ fils the est * "Ever companies that have made diverse recrultment apr ort il fal to break thisealture "arses. tn January 20¥7 "Bloomberg reported that although Facebook had stared ving recruiters an incentive to bring In more women, black, and ‘Latino engineering candidates back in 201, the program as ating ew new hires, Accordngto former Facebook recruiters, this was because the people responsible fr final hiring Culture Mist 25 Hiring Criteria “Hore Oy eo or eh approvals-twenty to thirty senior leaders who were slnost entirely whiteand Asionmen-stllassessedcandidatesby using the same metrics as always whether they ad gone to the eight school, already worked a top tech company otha frends at acebook who gavethema postive referral! Whatthismeans that, even after making i through round after round of iter views designed to prove thelr kil and merits, many diverse hires would be blocked at the inal stage—all because they det match the profile ofthe people already workingt Facebook, BREAKING THE PATTERN, Wa veiouseyle: these companies say they want diversity then tse exactly thesame wecrultingmethods they alwayshave, gingto exactly the same schools they've always sone to, and claim ‘there ae not enough highly qualified diverse candidates, When they do lind diverse hires hey expect them to remo them elves to fit the company’s existing culture~one that was tesigned for, and is reinforced by, «homogenous group. New hires who cantor wort do everything takes to become aul ture AU" leave-and the company convenient reinforces its texting ideas about which kinds of peopl tought to rer ects obviously hiring people who were different just didnt ‘workout, Over and aver aga, people ike Fatima, who are fen the most prepared to make product beter—to have diferent ideas, to call ut aps or problems, to identify where designers and engineers have blind spot~are pushed tothe sie "Thats why the pipeline is such a myth. Reardess of how many women and undertepresentod minorities study computer Telence th indasty will never be as diverse asthe audience its ecking to serve—aka all of wsiftech wot erate an ene onment where @ wider range of people fel supported, wel comed, and abeto thrive Thegoodnewsisthere+actallynomagictotech Asopague asitmightscem fromthe ousie,itsjustasillet—one tat ll Tuinds of people can, and do learn. Theres no reason t allo" tech companies to obfuscate their wor, to cll t special and fexcmpt Ht from our pesky ethies. Except that we've never demanded they do better. But we canand if we do, well not only make things beter forallthe Kayasand Fatimasofthe work, we'lalso makethings tetter for ourselves, every the we pik up our phones or open a browser ta, Chapter 3 Normal People AL Rt ters earl mina mo rom the Minneapalis suburbs? Ordo you see yourself asa “Matt the milenial urban dweller who loves Cross and co-brew coffe? Maybe you're more of a “Maria,” the low income community college student striving to stayin school while supporting her parents, No? Well, this is how many companies think about yo From massive businesses like Walmart and Apple to edging startupslaunchingnew app, organizations ofalltypesuse tools called personas—fctiona representations of people who St their target audlences-when designing thelr produets, apps, web sites, and marketing campaigns. Pertonasareoftenmesnttofellikerealpeople—sometimes right doven to Kellys 2014 Toyota Sienna (which she purchased ‘with her husband while she was pregnant with their second hk or Mates Phone Plus (which hejustreplacedbecausehe {dropped his last one outside the rock-elimbing ym). The spectficitycan be unnerving: you half expect to startearing about a pettonais childhood chicken pox o aversion to elanro, What fdoesthat have to do with how they usea website agin? “This level of speci ent added by accent 1 ams 0 sive personas enough descriptive deal and backstory to fee felatabletotheteamsthatuse thems tha, eal teammens ‘bers thinkabout them regularly and internalize their needs and preferences. Thats great in Uheory but when personas are crested by 2 homogenous team that has’ taken the time to understand the ‘nuances ofits aence—teams lke those west in Chapter? they often end up designing produets that alienate audiences, rather than making hem feet home, "Thats what happend to Maile Delano, She a PAD ca Alidate at MIT and an active participant in the Quantified Seif movement, loose organization of people who are interested in tracking everything from moods to sleep paterns to exer~ lac, One day in 2015, she decided to investigate tools for tacking something people have been monitoring for mille nla her period, Her eyele had been recent iregulasandshe wanted to doa better job of tracking both her period and her moods in relation to it. So she test-rove some menstrual ‘yee app, looking for one that would help her ge the infor ration she needed, ‘What she found was’ os Most ofthe aps she saw were splayed with pink and oral motifs and Delano immediately hated the gender stereotyping. ‘ut even more, she hated how often the product assumed that fertility washer primary concern~rather than, you know, 3k ing er (Gre Moyer ane Sa ‘Asa “queer woman not interested in having children” Del ‘ano found one app low, particularly problematic She wrote ‘The frat thing wae asked when I opened the app was lahat my “journey” wes: The choioes were avoiding peg nancy tying to conceive, o erty Westmont, And my “Jouney”iovlves none of thee Five aboard in ready tying to igrote the pp esuinotons that prog nancy ie why | want to tack my period. The app leo secunas that 'n sexusly ative with someone who can etme pregnantary Delano’ experience with Glow might have made sense bck in 2018, when Gla launched with he mission ofusng big data “to help get you pregnant” But in 201, the founders realized that abouthalfofGlow’s users were atuallyusingthe pp to avoid getting pregnant? So, with 817 million in new funding in hand, the team set ost to transform Glow from a narrow, fertility: focused experience toa product that could serve all women-inclading i would seem, women lke Del ano, “We live in a time when people ae tracking everything out their bodies. yeti’ sil uncomfortable to talk about ‘your reproductive health, whether you're trying to get pres ‘ant jst wondering how ‘normal your period isthe com: any website stated. "We believe thie eed to change"* And ‘the people who thought they were the ons to changeit? Glow's founding team Max Levehi, Kevin Ho, Chris Martins, and Eveby Giow snow app Seine eater Cow Ryan Ye. All men, ofeourse—men who apparently never com sidered the range of real people who want to know whether theirperiod e“normal™ Since Delano'sarticle,Glowhas actually updated its prod ucts and how it talks about them-repostioning Glow as an “ovulation calculator” and launching a separate app, Eve by tow, for period trackingand sexual health, Only one pecblem: Eve might offer the features Delano wants—it can track her Periods and her moods—butitstil makes aton of assumptions about its users, referring to them as “girl” using slang lke “hookups,“and describing sexina way that's centered entirely ‘onmale genitalie:a banana witha condom, a banana without a condom, or no banana. youre an adult woman in a relation shipwithaavone wt oo aman yout probably silgingta feel left out. nitese Technically Wrong WHEN “NORMAL” BECOMES NARROW “Thiskludotthinghappensall he time:compantes imagine their deaiod user, and then create documents ke personas to (desrbe them, Butonce you hand them out aa meeting o post then in the break room, personas ean make teasy for teamsto start designing onl for that narow profile. And itean happen renin atch company where women areon taf ike Btsy. Etsy san online marketplace for buying and selling hand made gods directly rom thle ereators—anthlng from eter press rectngcardstohand-knit baby booties to wood shelving nade fom salvaged barn wood, As you might gues, great place to shop for unique git "That's precsly what Etsy wanted Brin Able to doin Jam ary 2017, when they sen her an ler on er pone: “Move ver, om tey, Sve doer wont Acntne' Day te fren — fecparneraaworan (0 eet Cup” trea, “We've got what he want, Shop Valentines Day siftsor him.” But, a8 with Maggie Delano, Able partner int @ man, She's not buying anything for “hla on Valentine's Day Appar ently, Etsy’ designers and copywriters never thought about this-never considered just how many people they might alien- ate wit tls message. Aber was iritated, “Come on, what are ‘the ons we'llget away one’ Uh, 1008, she joked on Titer? ‘This sot of problem happens whenever a team becomes Iyperfocusedon one customer group, and forgets toconsidr the broader rangeof people whose neds could be served by ts prod tuck. In Etsy's eas, that oversight resulted in leaving out tons of eople—not just those inthe LGBTQ community, but also those who are single and might want to buy gift for loved ones... oF simply not be told they ought tohave him toshop for. And all Decause the team tailored its messages to an imagined ideal woman na heterosexual reltionship—without pausing twaskowho mightbe excluded, orhow itwould fee fr them. ‘Thats what we sw in Glow too. Eve by Glow works well for ten gris and young women who are sexually active with bos, Glow works wel fr women who are trying to get pregnant with 1 partner. But for everyone else, both services stop making sense—and can beso alienating that would-be users feel rus trated and delete them. NARROW VISION, NARROW DEFAULTS ‘Thiskindofnarrow thinkingabout who and what is normal also ‘makes its way into the technology isl, in the frm of default settings Defaults are the standard ways a sytem worke-suchasthe ringtone your phone islready set to when youtake tout fof the box othe fact thatthe "Yes, send me your newaletc!” ‘Guecbox comes preseestd nso many online shopping carts. These settings are powerful and not just because we might not note that a checkbox is already selected (ough you can bet marketers are relying on thad. Defaults also alfect how we perceive our choices, making us more likely to choose whatever is presented as default nd less ely to switch to something else. Thisis known asthe defo ct. ‘etwven the defi effect making ws more likely to value preslectedchoies andthe fact that many ofusetherdon ant to bother adjusting our settings or dott know tht we can, very few of ws tally change the default settings onthe systems we ‘ae, That's why youl hear the Phone Marimba ringtone ver ‘where you go (and see more than one person nearby check their huge and pockets, People who design digital products know this, ad some of thom we that fact to make money-Hke when New York City abs implemented touchscreens in every vehicle. The sereens ‘defnlted to show your fare and then afew options to automat ‘ally add the tp to your total 20 percent, 25 percent oF 30 pe ent, Average ips went from 10 pereent to 22 percent, because ‘he majority of riders-70 percent—opted to select one of the {efaut options, ater than doing thelr own caution ‘Deaults can also be time-savers fr users, One cou even argue thatthe tipping deft in New York taxis re just that, nce they ellow customers o skip the math when paying thelr fares hough, it would be hard to convince anyone tha'sall the designers had in mind, Or if company has primarily US cus towers, tight default o United States when users enter thelr wddress into a shipping form, xo that most users don't need to scroll through biglstt find thelr country Default settings an be hepfal or deceptive, thoughtful frustrating, But they're nevee neutral. They're designed. Ax ProPublica Journalist Lena Groogee writs, "Someone, some- ‘where decided what those defaults should ean it probably What happens when those someones are the people we met in Chapter 2: designers and developers who've been tld that they're rock stars, gurus, and genluses, and thatthe word sade or poplelike them? In 2015, middle-school student Madeline Messer found out Arsthand. Like many kidsherage Messer loves playing gamesion her phone, often alongside her fiends. One day, she noticed & friend playinga game usingaboy avatar. When Messer asked her why she wast playing as agi, her fiend replied tat it simpy ‘wasnt an option: only oy charatersesstd in the game. ‘This dd si well with Messer." started to pay stention toother apps my friends and I were playing” she wrote in the ‘Washington Post “saw that lot of them fetured boy charac ters and fir characters dd exit, you were actully requited topay or them” ‘With her parents’ permissian, Mester embarked on an experiment she dowloaded the top éfty “endless-runner ames from the Tunes Store and set about analysing their default player settings Endless runners ane games where play- ers aim to keep their characters runalng as long as posible, rackingupas many pontsasthey can before, eventually they it obstacles and are detested. Mosser found that nin out of these ity games used non6 Technically Wrong gendered characters, such as animal or objects, OF the remain taorty one app allbutoneolfleredamaleckaracter—butonly terony-thece of them, less than all, fered female character otions, Moreover, the default characters were nearly always Tate: Almost 90 percent of the time, payers could use a male ‘haracter fo fe. Female characters on the ater hand, were Include as default options only 15 pereent ofthe time, When female characters were avalablefor purchase, they cost an ver ge of $753 nearly twenty-nine times the average cost of the ‘origina app download. A similar dfvolt i play whenever you sign up for anew app oreweate an account on a website that uses profile photos, and you're automatically given a male avatar—the con of per avesithouette used by the system to depict anyone who hasnt tploaded a petue ye. In fet, thats how Facebook treated pro flee without an image, up unt 2009 ors, when a female ver tion was added tothe mix. Today, more sites are defauting to neutral avatart—eitherbymakingthesilhouttesrore abstract, ‘and therefore less gendered, o by sing some oter fen rep resent user such asthel initials "Wecai also sce default biases fn ation by returning tothe smartphone assistants I mentioned in Chapter Is Apple's Siti Google Now, Samsung’ $ Voce, and Microsofts Cortana, In ‘adiion to not understanding queries ike “Twas ape” these tervices al have another thing in common: women’s voices eve asthe defalt for each of them, As Adrienne LaFrance, ‘writing inthe Atlante, pt it, "The simplest explanation i that people are conditioned to expect women, not men, to bein ‘eiministrative ole” Gust think bout who you pieture when ‘youhwear the term secretary) ret’ ook once more at Snapchat In adtion to the s- called “alme-inspired” iter we saw earlier, the app is known for releasing ers that purport to make you preter, lke the popular "beauty" and “Bower crown” features. These filters smooth your skin, contour your face so your cheekbones pop, and... makeyou whiter!" Why s whiter the default standard for beauty? Wel that's complexeulturs question but dou its ‘ne thatthe thee white guys from Stanford who founded Sap chat ever thought about "These might seem ike small things but default settings can se up tobe a big deal-both for an indvidual usr lke Messe, snd forthe eulturest ange. Just lookst the requirements for for ‘matting paper inalimost any eallege ass Times New Roman, 12 points, But that wasnt the case uni relatively reenty namely, the 1990s, when Microsoft Word started shiping with ‘Times New oman as the deol font, Most peaple stuck to the eft and eventually that detaultbecame the standard Dealt style for yur freshman paper comparing the por- trapal ofheroism in The Ojaey versus Beowulf might not mat- ter much (Since the begining of time." trite opening sentence in every fon) But when default settings present one roup s standard and another as “specal™sueh as men por trayed as more normal than women, o white people as more normal than people of colt people who are already mar~ sinlized end up having tbe mast dificult time finding technol cy that works for them Perhaps worse, the lass already present inourcultureare let reinforeed ‘That's why smartphone assistants defaulting to female ices is 30 giling: it renfores something most of us already2 Tehcally Wrong hvestuckin the deepbits fur brains Women arcexpected to the more helpful than menfor example, to stay Tate at work to sss colleague and are judged more harshiy than men when they dont do." The more we rly on digital tool in everyday life the more we bolster the message that women are scktys “elpers”~strengthening at sssocition, rather than weaken: ing it. Did the designers intend thie? Probably not. More Mikey. they ust ever thought ou it THE MYTHICAL MIDDLE ‘Try to bring up all the people design teams are leaving out whether ts gay people buying for loved ones or women who ‘antto play games~and many in tech wil ely, “Thats just an ecige case! We cant cater to everyone!” “Edge cases aclatslc engineering term forscenarosthatare considered extreme, rather than typical It might make sense to void edge eases when you're adding features: software that includes every “wouldnt be nice... scenario that anyone has ever thought of quickly comes bloated and harder to use. ‘But when applied to people and their identities, rather than toa product's features, the term “edge case" is problematio— Tecause it assumes there's sucha thing a an “averago” user in thefes place IHturnsout there fantwe real edge cases. And Tdontmean that metaphorically, but sientileall:acording to Todd Rose ‘who directs the Mind, Bran, & Baveatio program atthe Har ‘ard Graduste School of Education, the concept of “average” doesn't hol up when applied to people. Tn his book The Bnd of Average, Rose tes the story of Lt. GithertS. Daniels an ale force researcher, who, inthe 1950s, 98 tasked with firing out whether fighter plane cockpits weren. slzed ght for the plots using ther. Daniels studied more than four thousand pilots and calculated thelr averages fr ten phys cl dimensions, lke shoulders ches, waist, and hips, Then he tookethat profil ofthe “average plo” and compared cach of his four-thousand-plus subjects to see how many of them were within the middle 30 pereent of those averages for all ten Alsensions ‘The answer wat zero, Nota single one ft the mold of "aver age" Rose writes: Even more astonihing, Daniels iscovered that you pioted out just ee ofthe on cimensione of ze—sey, eck ctcumference, thigh cicumfororce and. wrist rcumforonce-—lest tan 35 pr een af plots would be verge #20d on all tree dimensions, Danoles fens wor clear and incontrovertible. Thare was no auch hing sverage pl, youve actualy designed itt tro one ‘o,whatdldthe ai force do Instesdofdesigningfor the middle, itdemandedthatsirplane manufacturers designfortheextremes ‘nstad~mandatingplanes that fitboth those at thesmallest and the largest sae along each dimension. Pretty soon, engineers ound solutions to designing for these ranges, Including aust- able sets, fot pedal an helmet traps-the kings ofinexpen: sive features we now take for granted, (Our digital products can do thistoo.I'seasy enough task users which personal health data they Uke to track, eatherthan forcing them into a preselected set of “normal” interest Ire easy enough to make form feds secept longer character counts, rather than euting of peopl names (more ofthat in the nest chapter. But tooaften tech doesn't find these kinds of cheap soltions-the digital equlvalents of adjustable stats ‘beause the people behind our digital produts are so sure they ‘ow what normal people ae ike that thet simply nt Lok: ingfor them, Brie Meyer and T wrote about thi ln Design for Rea Lit callingondesigerstolet gp oftheirnarrow ideas about “worms peopl," and instead focus on thase people whose identities and ‘tuations are often ignored: people transitioning their gender presentation, or dealing with unexpected unemployment ora zing chronicles, of trying to leave a violent ex: We didn’t tall hese peop identities nd scenarios “edge cases," thovsh ‘Wecalled them sires cases "evasubile shift but webeliewe san importantone, When designers call somesne am eg ease they Imply tha theyre not Iimportantenough tocareabout-thatthey'reoutside the bounds ‘ofeoncern.Inconteast, a stress case shows designers how trong their works~and where tbreaks down ‘Thats what one design team at National Puble Tso is doing, During the process of redesigning the NPR News mobile ‘pp senior designer Libby Baweombe wanted to know how to ‘make design decisonsthat were moreinclslvetoadiverse audi ‘ence, and more compassionate to that audience's needs. So he Ted session to identify stress cases for news consumers, end used the information she gathered to guide the team's design decisions The result was dogens of stress cases around many ferent scenarios, suchas “A parton fing ancous because fai nb in the locaton where breaking news i oecuring + An nglh language lesmer wha struggling to understand + worker who ean only access news tom thei phone while +A peron who feos upset baceuse @ sory viggered their memory of waumatic vent [None of these seenarlos are what we think ofa “average” Yet ach ofthese i entirely normal: theyre seenrios and felings that are perfectly understandable, and that any of us could find turselves experiencna ‘That'snottosayNPRplansto customize ts design for every single station, Instead says Baweombe, isan exreae in see Ingthe problem space differenti: donttying stress casos helps us ce ho spectrum of er specially alot se gency. Steus cases help us design or tel ng ito consideration moments of sires, ser jauney that al outsoe of ou sel cxoumstances and assumptions." Puttingthisnewlenson the producthelpedthe dsignteam see all kinds of decisions diferent: For example, the old NPR [News app displayed all stores the same way: just heeding andatiny thumbnail image. Thisdesignis reat forskimming something many users rely on-but it's not always gret for knowing what you're skimming. Many stories are nuanced,