Best of Design Matters: Dr. Joy Buolamwini

0:00This episode of Design Matters originally appeared in July of 2025. It wasn't as easy as saying, okay, let's make more inclusive data sets. And when we have more inclusive data sets, we'll have more accurate facial recognition. But accurate systems can be abused. And so the analysis had to be not just how well does the technology work, but what kind of technologies do we want in society in the first place?

0:30From the TED Audio Collective, this is Design Matters with Debbie Millman.

0:38On Design Matters, Debbie talks with some of the most creative people in the world about what they do, how they got to be who they are, and what they're thinking about and working on. On this episode, computer scientist and digital activist Joy Boulamwini talks about her career and about facial recognition technology in the airport. You actually have the right to opt out, but most people don't know. And it's not surprising because you go there and they say, step up to the camera.

1:11Hey, y'all. It's Brittany Kelly, founder of Tribe Kelly. We're a family-owned brand built on a simple idea to create American-made pieces that feel as good as they look. Everything we do is proudly made in the USA from start to finish, with a focus on quality, comfort, and longevity. These are the kinds of pieces you live in, the ones that get softer with time and stay with you for years. If you haven't checked us out yet, come see what we're all about at tribekelly.com. Most people don't realize how much of their personal information is being bought and sold every day.

1:45Data brokers are making billions, pulling details about you from public records and the internet, then packaging and selling it, usually without your consent. That's how your information lands in the hands of scammers, spammers, even stalkers. It's why you get endless robocalls and why ads seem to follow you everywhere. That's where Aura comes in. Aura actively removes your data from broker sites and keeps it off. They also instantly alert you if your information shows up in a breach or on the dark web.

2:16But Aura goes beyond data protection. With one app, you get a VPN, antivirus, password manager, spam call protection, dark web monitoring, and even up to $5 million in identity theft insurance. All backed by 24-7 U.S.-based fraud support. Other companies might sell just credit monitoring or even just a VPN. Aura gives you all of it together at the same price competitors charge for just one service. Start your free trial today at Aura.com slash safety.

2:48Protect yourself now at Aura.com slash safety. Dr. Joy Bulamwini is a computer scientist and a poet of code who uses art and research to illuminate the social implications of artificial intelligence. She founded the Algorithmic Justice League to create a world with more equitable and accountable technology. Her MIT thesis methodology uncovered large racial and gender bias in the world's largest and most powerful technology companies.

3:24Dr. Joy Bulamwini is depicted in the critically acclaimed documentary, Coded Bias, of which she is now on a five-year anniversary world tour. And the documentary sheds light on threats artificial intelligence poses to civil rights and democracy. And she is also the author of the best-selling book, Unmasking AI, My Mission to Protect What is Human in a World of Machines. Dr. Joy Bulamwini, welcome to this very special live episode of Design Matters at the WBUR Festival.

4:01You know, sometimes dreams come true and I'm living one right now, so I'm so happy to be here. Thank you. Thank you.

4:10Let's begin with Baby Joy. Baby Joy, okay. You were born in Edmonton, Alberta, just as your father was finishing his PhD. And you've described yourself as a daughter of art and science. What do you remember most from that unique collision of your mother's paints and your father's pipettes? That's a great question. In the book, I say how my mother asked questions of colors. My dad asked questions of cells.

4:42And in that exploration, I started asking questions of computers. And so for me, I literally grew up with art and science as companions through my parents. And in Oxford, Mississippi, that meant going to my dad's lab and feeding cancer cells, looking at squiggles on computers, learning later these were graphs and so forth, flow cytometry and that kind of thing. And then my mom, I just thought every weekend you go to art galleries and pitch paintings. I didn't realize that's what really was going on.

5:14And so I grew up with those worlds and it felt very much like an invitation to be creative, whether through scientific inquiry or artistic inquiry or for me, playtime, right? You know, so I think that was a true gift that I see now in the way that I do my work as a poet of code. You spent your early years in Ghana before moving to Oxford, Mississippi. What were the values that shaped you most in those formative years of in-betweenness?

5:46I think the first thing is my first language being chi. And so I used to have I have all kinds of accents. I code switch often. But when I first came to the United States and I was in Oxford, Mississippi, they actually put me in speech therapy. Why? They didn't realize English was my second. They should ask more questions.

6:09You know, and I and maybe I really did need to be in there, but we can deconstruct that later. I recently returned back to Ghana last year after 30 years. I know I don't look that old, but it was three decades since I'd been there and it was the best homecoming I could have imagined. In what way? One of the things that I noticed is all of my relatives were looking at me and I was looking at them, but none of us wanted to be rude or stare.

6:41So we would like look at each other through reflections and like just catch an eye. So it felt like I was time traveling when I would see my aunts and uncles. I was like, oh, that's what my brother's going to look like in a few years. Or, oh, that thing I thought my dad was just my dad. I see it in all of his siblings. So that kind of way. And then also connecting with my young cousins, I think they range from about age eight to 26. So I'm on the older end of the cousins within a more immediate family.

7:12And so it felt like a very warm embrace, a long awaited embrace. Given you were so young when you left Ghana to go to Mississippi, did your family share stories about Ghanaian excellence or legacy that helped shape your own sense of possibility? Yes. I'm third generation PhD. So my grandfather was a dean of a school of pharmacy, Kwame Nkrumah University, science and tech. And so we have some Ghanians in the audience here as well.

7:47They went to good schools, too, even if it wasn't science and tech, which is all solid. And so growing up, it was it wasn't like, oh, be excellent or that kind. It was just around us as this is our family legacy. There's this value of education. There's this curiosity. Got a library card very early. And we used to live within walking distance from the Oxford Public Library. So my brother would roll me there in a wagon and he introduced me to the boxcar children and hardy boys and that kind of thing.

8:22I have this vision of him driving you back in the wagon with stacks of books around you. Well, it was so fun because right now the documentary Coded Bias were on world tour and we just did a screening in Oxford, Mississippi at that same public library. And on the stage, I used to watch magic shows and puppet shows. And then they were showing Coded Bias and my nieces were there. My brother was there. One of my childhood neighborhood friends was there as well. So kids I grew up playing with and eating honeysuckles and, you know, that kind of deal.

8:55So it was really a nice full circle moment. And then a friend I had in Oxford, England, where I later studied, she had now become a professor at Ole Miss. And she was there with her baby and her dog and her husband. So it was this kind of culmination of many strands, different elements of my life there in Oxford, Mississippi. So the Southern Twang had come out just a little bit, you know, to remind where I came from. But it was all good. It was a good time. There's a lot of kismet in your life and we'll get to that.

9:26As a teenager in Memphis, you were building websites to cover your basketball team's uniform costs, writing Java games, pole vaulting and skateboarding. What did technology represent to you then as you were forging a path as an athlete? Yeah, well, it was a means to cover my dues. So instead of paying for the sporting dues, it was, OK, let me make a social media, a social network for the track team.

10:00Let me make a website for the basketball team because I'm going to be warming the bench, right? I might as well contribute in some other kind of way. My first website was really for my Latin club. So I was part of the National Junior Classic League. As one does. And I spent so much time on that website. It had animations and I was coding in Flash. And it was, I used to always say it was the top 10 in the country. So it probably meant it was number nine, but whatever, you know.

10:30So that was a little bit of encouragement. And then I was so fortunate now that I look back at it. At the time when I was in Memphis, Tennessee, I had the opportunity to take three different computer science classes as a high schooler. And part of the reason was because of my teacher, Miss Jill Connell. And she had this one room kind of schoolhouse where on each three walls and then her wall was on the main wall, her desk. She would teach the first version of computer science, the first class.

11:04She'd do the second one and then you'd have the AP class. So I circled those walls, right, for three years. And that really gave me a great basis. And I was looking back at some of the data. And I think at the time I took the advanced placement computer science test, only five kids in the state took it. And so that I was in that specific school with a young teacher eager to make it work, you know, even if it meant having to come up with three different lessons in that same period was really, really fortunate.

11:41And I used to spend, the nerd I am, I would spend my lunch period there coding Lego robots. And later when I would come to MIT, I would work in the lifelong kindergarten group that helped create and establish the Lego Mindstorms as well. So I always shout out to Jill Connell and follow her Facebook posts and her kids now and all of that. Well, you just said that you were a tech nerd or a computer nerd. Definitely. But you were also a coder, a creative and an athlete.

12:14So how did your friend group view you? I was always in between spaces. And so for my parents, as long as I got straight A's, I could do what I wanted, right? So I was like, cool, I'll get the grades. And let me try basketball. Let me try track and field. I didn't actually want to do cross country. I was recruited to cross country. And then I remember running my first mile throwing up and the coach was like, I think you got this.

12:45Do I, though? So that's how I ended up doing three different sports in high school. But I really fell in love with pole vaulting. I know. The most of all of those things. And didn't you have aspirations to go to the Olympics? I mean, if you don't do it, you've got to do it, right? Aim high. No pun intended. I couldn't vault higher. I couldn't, you know, I was at the, you don't start doing a full pole vault. You start in the grass, then the sand, then you move to the pit. So before I'd even cleared anything in the grass, I was already saying Olympics, right?

13:20So that just tends to be my mentality to aim high. You attended the Georgia Institute of Technology for your undergrad degree. Yellow jackets. You, but you first considered majoring in international affairs and biomedical engineering. Yes. You eventually transitioned to computer science. What did you envision doing professionally at that time? Oh, I mean, there were so many things I wanted to do. My first career aspiration was to be a professional skateboarder.

13:51Did that come before or after pole vaulting? Before pole vaulting. So I started skateboarding when I was around 11 years old. I watched the Goofy movie. And in the Goofy movie, they made skateboarding look so cool. So I got like the cheapest skateboard you could ever get from Walmart. It had this purple cobra on the back. Later, I got a much better skateboard, but that's how I started. So when we moved from Oxford, Mississippi to Memphis, Tennessee, that summer, I started learning how to skateboard, which meant scraping my ankles a lot, mainly.

14:25So you wanted to be a professional skateboarder, but you were majoring in computer science. So this was when I was 12. And then the reason I decided not to pursue that career path is I was introduced to the gender pay gap. So I was looking at skateboarding competitions and I was looking at what the men got paid and what the women got paid. So the women's prize was like what the 13th place guy got in terms of first place. I was like, you know what, I might need to try something else.

14:58So that's when I got off the skateboard path for a little bit. I returned to it later in grad school. And do you still? Every so often. These aren't the right shoes, but if you have the right shoes, I'll try to pull out a 180 boneless. I can still do that from time to time. In your third year at Georgia Tech, you were working with a social robot named Simon. And your assignment with Simon was to see if you could have the robot engage in a social interaction with a human. You created a project called Peekaboo Simon.

15:32Can you talk about that project? Yes. So the idea with Peekaboo Simon was to program the robot to do a simple turn taking game, right? So you cover your face, you uncover your face, you say Peekaboo. So that's what I was doing. The problem is Peekaboo doesn't work if your partner can't see you. And my robot was not detecting my dark skinned face. So I'm looking at my code. I'm like, I think the code is right. So what's wrong?

16:02So I didn't have that much time. And I had a light skinned roommate, red hair, green eyes, pale skin. Oh, you're perfect. So I used her to test it and it was working on her. So when it came to do the demo, I just made sure somebody with light skin tested it. And so this was around 2011 when I was a junior at Georgia Tech. Your senior capstone project at Georgia Tech was an experiment with the Carter Center. And you traveled to Ethiopia to do that work.

16:33And there you piloted a health data system in Ethiopia that eventually reached 17 million people.

16:4617 million people. What type of data system and how did you do that? Yes, so that year, I think I kept going around saying, I want to make an impact in African nations as kind of my little tagline when people would ask, what do you want to do? I'm Joy. I want to make an impact in African nations. Oh, that's cute. So I was doing that. And there was an invitation at Georgia Tech to come to some, I think it was a tech and

17:16health summit at 8 a.m. Like 8 a.m. I don't know about this, right? But at that year, I had my show up, speak up stand up. It was just like my little mantra for the year. So I said, OK, I'm going to show up. So 8 a.m., I show up. I'm going to speak up. So I speak up. I talk about my little African nations thing. And it turned out there was an epidemiologist from the Carter Center, and they were looking for a software engineer to help them with some of their global health programs.

17:47And in particular, they did a program called Maltra in Ethiopia. And so that was their malaria and trachoma program. And they'd been doing it for a few years. And now it was time for monitoring and evaluation. So they wanted to go in and take surveys and things like that. The problem is their paper-based surveys had some limitations, especially when you're trying to put in GPS coordinates. So if you get one of those digits wrong, right, you might be in a lake instead of the space you're supposed to be in.

18:17So at that time, open source was very helpful because Android had been released. And you now had these Android tablets we could program. So the opportunity was to replace their paper-based way of assessing the effectiveness of these programs into a tablet, right, into this digital form. And so that's where I came in with my computer science skills. So I was like, hmm, surveys, you could view it that way. Or we're transforming data collection, right, so we can have real-time insights on the trachoma

18:51program at the time. So that's what ended up becoming my senior capstone. And we looked at their current system, transformed it to something that had taken about 30 to 90 days we could do in one day. And so that's why they wanted to continue to adopt it. But I realized global health wasn't for me because while I was in the field, I was making up songs about pit latrines and hygiene and things like that.

19:24And that just was not the culture of the global health people at all. I might need to try something else. But that's, yeah, that was one of the formative experiences for me because at that time, I was at Georgia Tech and I was gaining all of these technical skills. And yes, I was working on robots and so forth. But I also knew those robots weren't going to be in people's lives in a major way anytime soon. It might be sooner now, but then a decade ago, it wasn't happening.

19:57And this desire to make an impact with the tech skills I was gaining at Georgia Tech, that's why that year I kept saying, OK, I'm going to speak up, you know, show up, stand up, all of that. So stood up in Ethiopia. You then applied for and were awarded a Fulbright Fellowship, which took you to Lusaka, Zambia. Yes. There you taught young Zambians to make mobile apps. Yes. You partnered with iSchool Zambia and worked on the Zidu Pads.

20:27Yes. Did I pronounce that correctly? Yeah. Yeah. So tell us about that. Yeah. So after the experience in Ethiopia, I started asking some questions like, sure, I came in with this technology that I had actually coded most of it in my childhood bedroom in Memphis, Tennessee. So it was during summer break. And because of that, I made all of these assumptions about the technology and the context that weren't true. So I was like, like for what, for example? So for example, I assumed that the internet speeds would be comparable.

21:01And so one of the features of that data collection process was you could upload it to the cloud and then the people, wherever they needed to be, could get the data. But because it was very slow, we ended up realizing we needed to download the data actually on SD cards. So I needed to, here I am under MosquitoNet, like in Ethiopia, changing the code because we hadn't accounted for the difference in internet speeds and things like that. Is it true that the students thought that you weren't an instructor?

21:34They never thought of you as an instructor? They still don't think I'm an instructor. I take it as a compliment now. So given that had happened in Ethiopia, when I had the opportunity to do a Fulbright fellowship, I was thinking, okay, it's one thing for me to come in. Yes, I am of the continent, but I'm also in some ways a Westerner kind of parachuting in. What would it look like to actually equip people to create the systems, the tools that address their own local context?

22:04And so that's where the Xamrise initiative came to be. And that was my Fulbright project. And then along the way, we met local skateboarders and poets. So there are a lot of side projects. I also couldn't be directly paid due to Fulbright rules. So we would barter certain things. So with the founder of iSchool Zambia, I bartered a house for the time I was in Zambia to then do training and be compensated for the apps that the students made through the training.

22:41And then they provided office space for that. So this whole creative bartering because of the constraint of not being able to be directly paid, we did an Indiegogo campaign and raised money to get the laptops for the students to learn how to code because I asked. What's an Indiegogo campaign? Oh, does anyone? Good question. So it was an alternative to Kickstarter. So crowdfunding platform. And I wanted to raise, I think we needed $10,000 or something like that.

23:15And so that allowed us to get enough laptops to then work with the students. What I had noticed with the Zedupads when I was talking to the founder of iSchool Zambia, I said, well, who developed the software? Who developed the apps? And they're like, oh, well, we used Eastern Europeans. And I asked why. They're like, oh, we're not able to find the talent we need here. So I asked, okay, if I did a program to train, would we then be able to have apps on the Zedupad? And so that's what the Zamrise program ended up doing so that when it actually came out, they could say it was also built by Zambians as well.

23:56So you've talked about the poetry scene and the skateboarding community you found in Lusaka. What was the connection for you between creation, culture, and code? For me, it was like a way of life. And even now I ask myself, how can I live poetically and how can I be expressive? And so while I was living in Zambia and I was making friends and so forth, I was like, okay, yeah, what do people do for fun? Where do you go?

24:27I remember going to a slam poetry session about subsidies. It was really interesting for me, right, just to see how people were exploring their creativity and things like that. So none of my friends were surprised when they saw the skateboard shots while doing the Fulbright. And we're also delivering for iSchool Zambia and things like that. So I never have felt that I've had to separate those worlds. You can have the impact.

24:58You can do it with a little flair, hopefully a little bit of style and connect with the artistic communities that are there. And skateboarding is another type of art. Did you teach anyone in Zambia to pole vault? No. See, pole vaulting requires some equipment.

25:16You need the pit. You need the poles. The poles are very expensive. Even the get skateboards, I learned, was tricky. So they didn't actually have a skate park at the time. And I was asking them where they were getting their supplies from and they were getting it from South Africa. You then went to Oxford, where you made history by proposing the first roads service year. What gave you the audacity to reimagine what the roads experience could be?

25:46Um, so all the work I did in Ethiopia meant that I had the qualifications to get into a global health program. So I applied and the road scholarship covers two to three years. And so I'd used my first year to do a master's in learning and technology. And the second year, I wasn't exactly sure what I wanted to do. So I applied to global health and I got in and I realized I really don't want to do global. I told you the culture, like my pit latrine wraps were not hitting.

26:19It had not improved. The situation had not improved since then. And so I wrote a 40 page proposal for this year of service and shared the type of visa I would need to have. Right. I really went in and how it would build on the work I had done in Zambia and so forth. And so basically when I sent to the roads trust, they said, don't get your hopes up, kid. We don't change much here. I was like, okay, I'm going to do it regardless.

26:51And you guys should get the credit, right? Anyways, they decided to agree to it. And so that became the first official road scholar year of service. But many senior scholars told me they'd been doing years of service unofficially well before me. Have there been official years of service that have come after you based on that change? There have been, actually. And they tended to be other scholars who are really entrepreneurial. And I remember talking to some of the members of the trust and they were curious why not as many people had applied for it afterwards.

27:26And I was thinking, I mean, the profile of a road scholar tends to be those who have mastered the academic game but aren't taking many risks. So I'm not so surprised that if you had the option to get an MPP or an MBA from Oxford, pretty much guaranteed, or do a year of service, most people would go for the guaranteed option. So it tended to be the crazy entrepreneurs who were open to that kind of risk anyways, right, who were doing it at the time.

27:57What did you do during that service year? Oh, so that service year, I started something called Code for Rights. And so Code for Rights was thinking through the curriculum that had been created in Zambia and actually adapting it to an Oxford context and looking at what kind of problems or issues might we apply that process to. And we ended up focusing on sexual assault on campus and creating this first responders app as well so that survivors would know what their options were.

28:31Thank you for doing that work. It's an area that I have particular interest in. You were then awarded a research assistantship at the MIT Media Lab from Ethan Zuckerman. Yes. The director of the Center for Civic Media. Now, is it true that he told you that if what you're thinking of making already exists, go elsewhere? Oh, so that was one of the Media Lab professors. That wasn't Ethan. Our group was kind of the everybody come sort of group.

29:03We had a big, wide open table. And every Thursday when we met, we would always have random guests and some regulars, you know, retirees where they would come to our table as well. So Ethan was very open, but the summer before I was in the area for my friend's wedding and I thought, let me go check out the Media Lab in case I apply there. And so I met another professor who was like, yeah, if what you're thinking of already exists, this isn't the place for you.

29:33So what was really interesting to me about Ethan's group was we were in the future factory, but we were a bit of an oddball because we wanted to talk about problems now, right? The Center for Civic Media. So we were doing things like creating systems that would allow kids to track school lunches to see if the government was delivering on what they had promised, the Promise Tracker app, which was deployed all over Brazil. So it was a group that was a bit out of the mainstream.

30:06But then in the time I was there, 2015 to about 2022, there was also quite a bit of change happening in the U.S. where people were saying, wait a minute, maybe the research we're doing, maybe we should be thinking about its immediate real world impact, right, with the different elections and so forth. So I remember in 2016, November 2016, after Donald Trump was elected, it was really interesting in my own circles because I'm from the South, right?

30:41So some people are like really rejoicing and other people are devastating and I'm seeing all of that happening. And at the Media Lab, a lot more people were asking questions of, okay, if this is happening, should I really be painting walls with my smile as my project or maybe there's something else I should be doing? And so I remember the day after the election, people were suddenly looking to our group. The oddball group that's talking about problems now became the mainstream.

31:12And that's around the time I was also starting to explore the work that became the Algorithmic Justice League. And so that was a transformation from being at the margins in the future factory to being a group that was viewed as a leader within the future factory. You signed up for a course that changed the trajectory of your life. Yes. Science Fabrication. Yeah. This was specifically focused on building fantastical futuristic technologies. What did you begin to build?

31:43Yes, this is where I started to explore this idea of shapeshifting, right? And so we mentioned earlier, I'm from Ghana. I was inspired by stories of Anansi the spider, trickster spider that could also shapeshift. And I wanted to do the same, but I had a six-week deadline. And so instead of shifting my shape, right, I decided to shift my reflection in a mirror. And so in that process of figuring out how do I make it look like there's a mask on my face through a mirror,

32:16that's when I started to experiment with computer vision. So first you have to detect the face. So once I got that going, I started putting different people's faces on my own. So Serena Williams was one, looked like the greatest of all time. Then I thought, okay, this is kind of giving me theme park where you have the cutout hole and you put it through. Wouldn't it be cool if when I moved, it moved just like that? Anyway, so when I was moving, it's moving. So to do that, I needed a camera.

32:47So I put a camera on top of the mirror and then I needed software that would actually follow my face in the mirror. So I went online, downloaded some software that was supposed to help me do that, but it wasn't working. Just like my robot, my peekaboo robot wasn't working. I was like, huh, I think something might be off. And at the time I was experimenting, it was around Halloween time and I had a white mask for a Halloween party. So when it wasn't working, I started just experimenting with things in my office.

33:20I even drew a face on my hand, held it up to the camera and it detected the face on my hand. So that's when I was like, okay, anything is possible. So I reached for the white mask and before I even had it all the way over my face, it was already detecting the white mask. So here I was at MIT, this epicenter of innovation that I dreamed of coming to since I was a little girl. And I'm in white face to be seen, coding in a white mask at MIT. And so that was when it switched from, can I shapeshift like a Nazi the spider to hold up, side quest.

33:58What is going on here? You've said that while the white mask episode was disheartening, you didn't want people at the time to think you were making everything about race or being ungrateful for rare and hard won opportunities. And you felt that you felt that speaking up had consequences. What kind of consequences did you fear? Retaliation, being blacklisted, being kind of marginalized with the way my research was perceived.

34:32And so I remember even when I started exploring the research, grad students warned me. They're like, you know, X, Y, Z, they try to study bias, didn't go well. It's like, these are the bones of grad students past. That shadowy place, it looks like studying discrimination, right? Like places the light doesn't touch for a career. So I was highly discouraged from doing this sort of work because it was touching on bias and discrimination.

35:02So that was part of one thread of being discouraged. But another thread of being discouraged was like, this involves AI. AI involves a lot of math. I'm like, and, right, you know, engineering background, all of that. So the math wasn't an impediment to me, but others' perception of the type of work I was capable of, I was also seeing that come out in some of other people's perceptions. And so for me to actually pursue this research was going against all of the wisdom that was being passed down.

35:41My supervisor, he wasn't against the exploration, but he was really practical. You spent a year working on a completely different project. This is a two-year program. You want to do a new project halfway through. That might be difficult. Maybe this is a side project, right? And so I think all of the advice was well-meaning. And everything they warned me about did happen, right?

36:09So I guess this is where it helps to be stubborn. But also, I think the other thing for me was, even though the people around me at the time didn't quite see the vision, it still felt important to me. And I have to commend the Media Lab for creating a space where I could explore a sandbox, even if they didn't understand the shapes I was building. They're like, you do you, right? We don't know what you're doing, but figure it out. And there are many spaces that I wouldn't have even been able to explore that at all.

36:44You stated that in some ways, you went into computer science to escape the messiness of the multi-headed isms of racism, sexism, classism, and more. And those signs indicated otherwise, you wanted to believe that technology could be apolitical. Oh, 100%. So what changed your mind about speaking up? When I saw the reaction to the 2016 election, it was kind of an all-hands-on-deck moment.

37:20And this questioning of what can I do to make a difference in the world? And so when I got to the Future Factory, there were two missions. One, get the PhD, family legacy, third generation. I only applied to, I didn't really want to go to grad school again, like, by this time, you know? So I kind of did the, well, I'll apply to one place, and if it doesn't work out, I did try. So I applied to one grad school.

37:50It worked out. So this was the price for getting it. Did you really think you weren't going to get in? To the media lab? Yeah. You never know for sure. I thought I had a good chance, but it wasn't 100%. And if it didn't work out, I could continue my entrepreneurial dreams. So I was not, I was not heavily invested in getting in. In fact, I was going to be working on an entrepreneurial project until I got a call from my father. It's like, remember who you are? Oh, no.

38:20I was like, Rhodes Scholarship Fulbright Fellow Daddy. This has to count for something, right? You know? You're like, it is not a terminal degree. Okay, okay. All right. All right. That is true. You know, so I applied to that one place and got it. Yeah. You think you'll ever go back to that entrepreneurial project? It's a whole other area of questions that I'd love to ask you about. But again, the time is slipping away. Yeah. I mean, I was working on a hair care technology company. I'm a CTO of an ed tech company.

38:51The hair technology company. We were about a decade early on hair analysis. But now if you go to my Havana, we can actually do a hair analysis where you get a camera. It looks at your hair strands. It can tell you so many things about it, but also give you unique product recommendations. So I was a CTO for that before I went to Zambia. So I had a lot of side projects.

39:17This episode is brought to you by Lexus. Have you ever considered that the most powerful tool in design is knowing what will move us? Design isn't just about function. It's about feeling. And it's often the smallest details that shape our experience. At Lexus, every car is intentionally designed with that in mind. Master artisans called Takumi don't just focus on what you can see. They obsess over what you can feel.

39:47The way the door closes with just the right weight. The precise click of a switch. The acoustic feedback of every control. Instead of relying solely on instruments, Takumi artisans add something more. They rely on human perception and intuition, honed over years to create experiences that sound and feel just right. That's the philosophy behind Lexus. Engineering to be felt. Because a car that doesn't make you feel something is a car that's stopped short of amazing.

40:23Experience amazing at your Lexus dealer. Hey y'all, it's Brittany Kelly, founder of Tribe Kelly. We're a family-owned brand built on a simple idea to create American-made pieces that feel as good as they look. Everything we do is proudly made in the USA from start to finish with a focus on quality, comfort, and longevity. These are the kinds of pieces you live in. The ones that get softer with time and stay with you for years. If you haven't checked us out yet, come see what we're all about at TribeKelley.com.

40:58Finding great candidates to hire can be like, well, trying to find a needle in a haystack. Sure, you can post your job to some job board. But then all you can do is hope the right person comes along. Which is why you should try ZipRecruiter for free at ZipRecruiter.com. ZipRecruiter doesn't depend on candidates finding you. It finds them for you. Its powerful technology identifies people with the right experience and actively invites them to apply to your job. You get qualified candidates fast. So, while other companies might deliver a lot of hay, ZipRecruiter finds you what you're looking for.

41:34The needle in the haystack. See why four out of five employers who post a job on ZipRecruiter get a quality candidate within the first day. ZipRecruiter. The smartest way to hire. And right now, you can try ZipRecruiter for free. That's right. Free at ZipRecruiter.com slash zip. That's ZipRecruiter.com slash zip. ZipRecruiter.com slash zip. Your thesis, which you titled The Gender Shade Project, demonstrates the priorities, preferences, and biases of those who create code and the algorithmic bias from companies, including IBM, Amazon, Microsoft, and more.

42:15With over 3,400 citations, you exposed how AI facial recognition systems had 100% accuracy for white male faces and near coin flip results for dark-skinned women. Some labeled Michelle Obama as male. Others were labeled as gorillas. What were some of your other findings? Yes, so there are a whole combination of findings when it comes to the mislabeling of faces.

42:51So with the Gender Shades Project, as I started looking at how computers read faces, I was asking three different kinds of questions when it comes to facial recognition technologies. So the first question is, is there a face? This is face detection. So when I'm putting on a white mask, that's a face detection fail, right? Another kind of question I like to say you might ask is, what kind of face? So what's the gender of the face? What's the age of a face? Maybe what's the emotional expression on the face?

43:23And so that's what I focused on for my master's work with Gender Shades. I was looking at companies guessing binary gender. And in that, we tested a number of different companies. So in the case of Microsoft, it was that there was perfection for one group, the lighter males, the pale males, affectionately called, right? And it wasn't so great for other groups like darker females. But it was also interesting because we tested like a company from China.

43:55We found that it actually had best performance on darker male faces. And this was really important because in that research, we were doing intersectional analysis, barring from Kimberly Crenshaw's work on discrimination, right? Along multiple axes. I was like, oh, this is being applied in the legal space, but maybe there's something for computer scientists to learn. So instead of just looking in this case at gender, I also started looking at skin type as well.

44:26And so it wasn't just the story of, okay, it works better on male faces than female faces, which was the overall trend. Or it works better on lighter faces than darker skin faces. But when we did that subgroup analysis, right, lighter males, lighter females, darker males, darker females, that's where we got the stark contrast you were just mentioning. And Microsoft was the good results, right? With IBM, their error gap between their best performing group, lighter males, and their worst performing group, darker females, the highly melanated like myself, right, was around 34%.

45:02So those were the findings that really got me to start exploring what are other ways in which computer vision has failed, right? And so you have the Gorilla Gate example that you just brought up with Google Photos, where what I now call an evocative audit, people in the wild, regular people were interacting with these systems and seeing issues. Google fixed the problem not by making a better system, but just by removing any label of Gorilla.

45:36So Gorilla's were also not labeled Gorilla's, just to be extra safe.

45:41I don't know if that's quite the solution. But so we've seen an evolution of different approaches to addressing some of these misclassifications and mislabeling that comes on. But as I was also doing the work, I realized even if these systems were perfectly accurate, whether we're going from guessing the gender of a face, which that's difficult, right? How does a person identify to figuring out the unique identity, right? With facial identification. If you have perfect facial identification and cameras everywhere, we have the infrastructure for a surveillance state that tracks your every move, where you go to worship, who you see at night or in the morning, whatever your preferences are, right?

46:27Where you go and protest. And so that was very interesting tension for me to hold as I was doing this research, because it wasn't as easy as saying, okay, let's make more inclusive data sets. And when we have more inclusive data sets, we'll have more accurate facial recognition, but accurate systems can be abused. And so the analysis had to be not just how well does the technology work, but what kind of technologies do we want in society in the first place?

47:03Amazon dismissed your findings. How did you find the confidence to stand by your data? It was interesting because we were talking about my parents earlier, right? So we did our first research paper, Gender Shades, and when it came out, we had tested IBM, Microsoft, Face++, this company in China. Our second paper tested those companies, but also included Amazon and another company.

47:36So when the first paper came out, it was actually fairly well received, right? Like, okay, bias is an issue. We're working on it, or some companies, like, we've been known, and we've been working on it. I was like, okay, okay, but this was the approach. So when the second paper came out, the reaction was surprising to me and to my research team, because imagine having a test where the answers have been known for a year and still failing, and you're a company as big as Amazon.

48:08And you're going for a billion-dollar contract with the Pentagon, which was happening at the time when this research came out. And so we're showing their competitors that are also bidding, right, are performing better than they are on this computer vision task. And so I was surprised at the Amazon attack, because the pushback I had expected from the first paper didn't come. But this is what the grad students of the past, right? You know, the places you shouldn't go.

48:38The bones. And the bones, right? This is what they had warned of. And oh, yeah, it came down hard. I remember I was spending my birthday in Switzerland, but I was at a conference. I was at the World Economic Forum, and the paper was going to be released about a week later. And so I remember flying from Zurich to Honolulu. I'm in this big red jacket, and I come, I'm like, okay, I need some Hawaiian shorts, right? And I had sent the pre-results to all of these reporters, and Amazon was claiming they had never seen the paper, even though I had a, you know, I had the email evidence.

49:20Maybe it hit spam. I don't know. Benefit of the doubt. Yeah. Maybe. Right? So they did all of these delay tactics. And then when the paper came out, they had a corporate vice president, Dr. Matt Wood, come out and try to discredit the paper. So much so that we had the research community rally around us and say, this paper is valid and important because the types of issues it addresses, we have to all be thinking about as AI becomes more entrenched in our lives.

49:54So we don't get better as a field by ignoring the problems. We have to actually confront it. And so this coming from a Turing Prize winner, you know, our Nobel Prize for computer science, and a former principal AI researcher of Amazon really had a major impact. What I found interesting, though, is when the book Unmasking AI came out, it was an Amazon best books pick after all that time.

50:28And even as this was going on, I separate individuals from institutions. There were people inside the company sharing what they were able to in terms of the opposition that was actually inside the company about the use of facial recognition. And I am happy to say that all of the U.S.-based companies we audited stopped selling facial recognition to law enforcement. Thank you. Thank you. Very important work. Let's talk about the other side of facial recognition that you mentioned and the whole sort of way in which we're now being monitored.

51:07How are you? What's your feeling on all of this? I think we have to be very careful now. I was just looking at the recent announcement with Sam Altman and Johnny Ive and thinking about what future computing devices might look like. And if you look at companies like Limitless.ai, Limitless.ai, which is a pendant that's always on, always recording.

51:39I'm thinking, where's the consent piece? So some years ago, Google experimented with Google Glass, right, where you could record and so forth. And people didn't like it. They're calling it glass holes and things like that. But I was reading on LinkedIn an experience from Allie Miller, who's this top voice in AI. And she shared how she was backstage having a conversation with somebody and said, oh, we said so many good things.

52:11I wish we had written it down. And the person said, oh, don't worry. I have an AI recording it right now. And so that level of violation of privacy consent is going to only increase if we don't push back. One of the ways we've been pushing back with the Algorithmic Justice League is our Freedom Flyers campaign. So you have the TSA has been putting facial recognition at the checkpoints in airports. They have plans to expand it to 400 airports.

52:45And you actually have the right to opt out. But most people don't know. And it's not surprising because you go there and they say, step up to the camera. There's no signage that says you have the right to opt out. It's usually very difficult to see. So we've been doing surveys to see if people are actually finding the signs. And oftentimes people don't even see the signs. Last year, we talked to the Department of Homeland Security. We showed our data. And so they committed to the spring actually adding language on the kiosk to tell people to opt out, which is a good step.

53:19But we really have to think about the power dynamics. You're in line. I don't know if you're early or late to the airport, but usually I have time pressure. You might not have time pressure, but I have time pressure, right? There's social pressure, people behind you in the line. There's financial pressure. I don't know how much your tickets cost, but the prices seem to keep going up, right? So you're doing all of that. You finally made it to the airport. Now they got the scan thing going on. And a TSA officer tells you to step up to the camera. They don't tell you have the right to opt out.

53:51You've seen everybody else in front of you step up. The signs are there. They're usually turned around in a different language. We've documented all the things. They managed to put the part that says you opt out right where the pullout piece goes. It might have just been a coincidence. You know, we've been documenting some of these things, right? So it's not designed so you're actually aware. And one thing they know how to do at airports is design signs, right? No weapons formed against TSA shall prosper.

54:22It's a close to $15,000 sign. Real ID. You need to have that. The signs they want you to see, TSA is hiring. You will be able to see those. So, you know, there's all of that going on. So the big thing we've been doing is, one, letting people know they have the right to opt out. Because we're getting this story that, well, no one refuses it. They don't know they can. But the other thing we've been doing is also collecting travelers' experiences. And some of them have been really disheartening.

54:53I remember reading one about a son, the son of a man, and he had autism and his son was scanned against his will. You know, I told my parents to opt out because of the work I do. And I remember them being humiliated in line, right? And so when I sat down with Secretary Mayorkas at the time, I shared that story. But the stories of many other people who talked about intimidation tactics or just being dismissed as like, well, we have cameras all over the airport anyways.

55:27I just came back from Greece. And this was the very experience I had when I was crossing over when I said, oh, I want to opt out. Smug look. They're like, we have cameras everywhere. This doesn't matter. But this actually does matter. And why it matters is if we don't opt out and if we don't resist, then that narrative persists that people want this, which they'll use. And they claim right now that they only put the facial recognition on some of these checkpoints and that they don't have it on the overall system.

56:00But all of this is just one line of code from being mass surveillance if we don't actually resist. And so that's why it's important, even if you haven't opted out in the past, that you continue to do it in the future. And so then going back to that situation where you're backstage and someone has their AI device, right? If we want a consent culture, we have to practice a consent culture and also demand a consent culture. Because what we will see if that doesn't happen is listening devices everywhere, which we are already set up to do, right, with our cell phones, but even more ubiquitous and even more intimate.

56:42Aside from not providing consent, what else can we do to combat mass surveillance in our society? Yeah, I think this is where there's individual action and collective action that's needed as well. At the end of the day, it does come down to laws, right? And so seeing, for example, the EU AI Act, where they have a restriction on the use of live facial recognition, shows that there's alternative ways.

57:14Now, it does depend on the administration. What we saw with facial recognition, we still don't have federal laws around it. But we were able in 2019, 2020, to get quite a bit of traction at the municipal level and also at the state level. What's going on in Congress right now, and now at the Senate, is there's a push to say any kind of regulation of AI at the state level is going to be on pause for a decade.

57:47That's huge. That's saying the protections that people fought for, because at the federal level is a bit harder, but at the state level, we can get traction. All of that gets rolled away. So that's what we should be putting our eyes on right now. Given the current administration, what are your hopes and fears about what is possible from a collective public response? I think my fears right now, especially with the gutting of people in the administration or people at the IRS, people at different federal agencies, is this assumption that humans are so easily replaced.

58:30So I think about this is why stories matter so much. It's not even the power of AI, but the power of the stories we tell about AI. So I think about ANETA, the National Eating Disorder Association. They bought into the hype, right? AI is going to be better than humans. It can do call center work, etc. And at the time they had their workers were trying to unionize, you know, for the reasons you do, better pay, better conditions, all of that. They decided to fire all the workers and replace them with AI.

59:01So you had a headline, right? Net of fire staff replaces with AI. I don't think it was even a week before you had the next headline, which was the chat bot that the people were replaced with had to be shut down. And that chat bot was actually giving people with eating disorders advice that's known to make eating disorders worse. And so here we're looking at the actions of FOMO, fear of missing out. We have to adopt these AI solutions and also how easy it is to dismiss the work you're not proximate to, to think it's so easily automated.

59:36So now coming back to the government, right, we're seeing this clear out of humans and then we're going to see this rollout of AI systems as a replacement to the humans. And then we're going to find out, oh, we didn't need humans after all, right? And we're going to be on backtrack. Surprise, surprise. Tell us about what you're currently doing, because you are on the front lines of helping society manage all of these issues and protecting society from the issues.

1:00:09Talk about what you're currently doing. I'm going to ask you one last question and then I'm going to have to close out. Oh, it's been so much fun. I know. OK, so what I'm currently doing is going global. I mean, not that the U.S. isn't a great place to be right now. Sorry, allergies. So last year, Oxford University, where I did the Rhodes Scholarship and did a master's in learning and technology and didn't do that master's in global health.

1:00:40Two master's is enough, right? Not for my day. You don't have me out here, out here working. So they reached out and they started a new institute focused on ethics and AI and they had a fellows program. So they asked if I'd be an inaugural fellow. I was like, oh, what does it entail? Well, tell us what you want to do when you want to do it and what resources you need. Is that a blank check?

1:01:05Sound like a blank check. I'll take I'll take the blank check. And so long story short, what ended up happening is I proposed doing a five year anniversary world tour of the documentary Coded Bias. Coded Bias premiered at Sundance in 2020. And when I watched it in 2025, it was as relevant as ever. We're talking about issues of bias and discrimination. AI still persistent and people see it more now. Right. Because if you are applying for a job, you know, you might not get it.

1:01:39If you're applying for a loan, all of these areas now, AI is becoming a part of the decision making system. If you're applying for college, if you get the right medical diagnosis, if the transcript of what happened when you were talking to your doctor is even accurate. All of this is now being mediated by AI systems. And not to even talk about. I mean, there's so much to talk about regarding the current potential situation of the administration looking at social media to allow you into the country or not.

1:02:12Absolutely. And you've already had issues in the past, right, where context collapse happens. So someone might say she's the bomb. Was that bomb? Was that a bomb threat? Right. Like this is real. And that is there are two parts. Right. One when it's mistaken and one when it's intentional. Right. As well. We've talked so much about your science, your innovations, the way in which you're working to protect society. We haven't talked very much about your poetry.

1:02:43And for those that are interested in seeing more about Dr. Joy, you can go to poetofcode.com. Before we close out the show, you have some really beautiful poetry in your book, Unmasking AI. You've done extraordinary performance art. I'm wondering if you can share one poem from your book with our audience today. Only one? We don't have time for more? I'm up if you are. I don't know about everybody else in the production team. Okay, I'll do two, if they permit me, because I want to end on a happy note.

1:03:18But before I end on the happy note, that's the note I think we should all hear at the moment. So this poem is called Unstable Desire. It's in the epilogue where I talk about my roundtable discussion with President Biden. Unstable Desire. Prompted to competition, where be the guardrails now? Threat in sight will might make right. Hallucinations taken as prophecy. Destabilized on a middling journey.

1:03:48To outpace, to open chase, to claim supremacy. To reign indefinitely. Haste and paste control altering deletion. Unstable Desire remains undefeated. The fate of AI? Still uncompleted. Responding with fear, responsible AI beware. Prophets do snare. People still dare. To believe our humanity is more than neural nets and transformations of collected muses. More than data and errata.

1:04:20More than transactional diffusions. Are we not transcendent beings bound in transient forms? Can this power be guided with care? Augmenting delight alongside economic destitution. Temporary band-aids cannot hold the wind when the task ahead is to transform the atmosphere of innovation. The android dreams entice the nightmare schemes of vice. Poet of Code. Certified. Human-made.

1:04:52Thank you. We said you're going to read another one. So this is one I wrote in the chapter called Intrepid Poet. And this is when I start flexing the storytelling muscles. Because for so long I felt that if I let my artistry out, my research might not be taken as seriously. And it was such a relief to find that it was moving from performance metrics to performance arts that really moved this message around the world.

1:05:24So to the Brooklyn tenets and the X-Coded, resisting and revealing the lie that we must accept the surrender of our faces, the harvesting of our data, the plunder of our traces. To the Brooklyn tenets and freedom fighters, we celebrate your courage, no silence, no consent. You show the path to algorithmic justice requires a league, a sisterhood, a neighborhood, a podcast. To the Brooklyn tenets and freedom fighters around the world, persisting and prevailing against algorithms of oppression, automating inequality through weapons of math, destruction.

1:06:19We stand with you in gratitude. You demonstrate the people have a voice and a choice. When defiant melodies harmonize to elevate human life, dignity and rights, the victory is ours.

1:06:41Dr. Joy Bolamwini, thank you so much for making so much work that matters. And thank you for joining me for this very special live episode of Design Matters at the WBUR Festival. To know more about Dr. Joy, you can read her book, Unmasking AI, and see more of what she's doing on her website, PoetofCode.com. This is the 20th year we've been podcasting Design Matters, and I'd like to thank you all for listening. And remember, we can talk about making a difference, we can make a difference, or we can do both.

1:07:14I'm Debbie Millman, and I look forward to talking with you again soon. Design Matters is produced by the TED Audio Collective by Curtis Fox Productions. The interviews are usually recorded at the Masters in Branding Program at the School of Visual Arts in New York City, the first and longest-running branding program in the world. The Editor-in-Chief of Design Matters Media is Emily Weiland. Hey, y'all, it's Brittany Kelly, founder of Tribe Kelly.

1:07:45We're a family-owned brand built on a simple idea, to create American-made pieces that feel as good as they look. Everything we do is proudly made in the USA from start to finish, with a focus on quality, comfort, and longevity. These are the kinds of pieces you live in, the ones that get softer with time and stay with you for years. If you haven't checked us out yet, come see what we're all about at tribekelly.com.