Transcript 0:00 [upbeat music] AI image generators have to some extent taken over social media. We've all seen them, the AI images of how people would look 30 years older or younger or what their kids would allegedly look like. 0:13 And what about that crazy couple of weeks where everyone posted AI-generated posters of themselves in Pixar movies? Well, in Brazil, that trend took a controversial turn last year. 0:24 Renata Souza, a state lawmaker in Rio de Janeiro, sought to generate one of these images for her Instagram page. 0:31 The prompt, she says, was for a Black woman with an afro haircut with African pattern clothing in front of a favela backdrop. 0:40 The AI generator produced an image of a woman holding a gun, prompting the ignition of debates about whether or not the algorithms that govern these tools, and which are present in our daily lives, are in fact racist. 0:52 I'm Ewan Marshall, deputy editor of The Brazilian Report, and this is Explaining Brazil. 0:57 [upbeat music] If you like Explaining Brazil, you should subscribe to The Brazilian Report, the journalistic engine behind this podcast. 1:09 We're an independent organization funded by subscribers, and you can help us stay independent and continue to produce award-winning journalism. 1:19 And if you are already a subscriber, you can go the extra mile and join our Buy Me a Coffee fan page. 1:27 In return, you will get exclusive perks like special newsletters and behind-the-scenes content, as well as a shout-out here on our podcast. 1:36 And today I want to thank our Buy Me a Coffee members, Carson Allen, Gabriel Lucca, Andrey Novoseltsev, Ken Ludwig, Leslie Seo, Mark Hillary, Luis Hans, Aaron Menes, Aaron Berger, Lars Vresvik, Alasdair Townsend, Müller Renascido, Peter Abrahamson, Jim Awofadaju, David Dixon, José Rosa Stankovich, Emerging Market Muser, Anna Lund, Peter Suffering, Anderson da Silva, and someone who chose to remain anonymous. 2:09 And our Buy Me a Coffee members come from all over the world, so please, if we are butchering the pronunciation of your name, do send us an email. 2:18 And if you too believe in the importance of independent journalism and want to hear your name on our podcast, go to buymeacoffee.com/brazilianreport and subscribe to one of the membership levels. 2:30 Click on buymeacoffee.com/brazilianreport to learn more. 2:36 [upbeat music] To discuss the phenomenon of racism within algorithms, we're joined today by Tarcisio Silva, senior tech policy fellow at Mozilla, who literally wrote the book on the subject, entitled Algorithmic Racism: Artificial Intelligence and Discrimination in Digital Media. 3:00 Tarcisio, thanks a lot for being with us today. As we've mentioned, you've done a lot of research and writing on this idea of algorithmic racism. 3:07 If you could just start by just explaining what that is and maybe giving us a couple of examples as to how this manifests itself in our daily lives. 3:15 Algorithmic racism, uh, in my point of view, it's the political layer, if I can say like that, of algorithmic discrimination in several layers of life. 3:29 There are some specific cases that are more evident and they are more clear, especially around problems with visual computing, like systems not recognizing non-white faces in softwares like, uh, social media filters, 3:52 and also, uh, things related to, uh, law and security, like, uh, recidivism algorithms that can, 4:06 uh, have discriminatory, uh, rankings or results considering the communities that, uh, that people are from, especially around marginalized communities. 4:20 So in countries like USA and Brazil, this frequently means, uh, Black communities. 4:27 And one of the examples that a lot of people in Brazil heard about involved state lawmaker Renata Souza, which I mentioned at the top of the show. Could you tell us a little bit more about that case? 4:36 Uh, Renata Souza is a legislative, uh, state legislative councilwoman in Rio de Janeiro, and she and, uh, a- an advisor of her team tried to publish a AI-generated, uh, drawing, like a Pixar drawing, of her, a Black woman, in a favela setting, like a slum setting in Brazil. 5:07 And the person asked the software to represent her, a Black woman, in a leadership position in the street of a favela. But the software, uh, represented her with a gun in her hands. 5:22 So the software reinforced stereotypes around, uh, Black people in marginalized communities in Brazil. 5:31 This got a lot of, uh, media and reports and also some controversies because she, uh, talked about this case in Twitter, and a lot of people dismissed her claims and tried to dismiss her experience, including with offensive and racist remarks. 5:55 But there are a lot of cases similar to that in Brazil and in other places. One example that I can mention, and it's included in my book- It's from a, a illustrator called Gabriel Jardim. 6:14 And when Lewis Hamilton, the driver, was in Brazil, he drawed, uh, a, a small, uh, homage to him, uh, showing him and a Black child in a favela in Brazil. 6:31 And Instagram did not allow it, him to, uh, to advertise this image because supposedly the image had a gun or had a weapon in the image, and this was not true. 6:49 And it was an example of how social media platforms, even before the current boom of generative AI, can discriminate images of Black people, like producing images with stereotypes, like in the case of Renata Souza, or even putting and imagining stereotypes and violence, uh, images when they do not exist, like this case of, uh, of this illustrator, Gabriel Jardim. 7:22 Yeah, that's really fascinating. And there's one thing which I'd like to understand better. It's regarding how an algorithm becomes racist. I mean, how do you make an algorithm racist? 7:32 Yeah, I think this is the, the main question, the quick question to, uh, talk and mitigate algorithm racism. Because it's, it's not that the algorithms are racist in itself, but the problem 7:50 can be, uh, associated to different reasons. The main one, and the most discussed one, is the, is a biased data or a biase- biased data set. 8:08 So when a system, a model, is trained with biased data that represents, uh, discriminatory societies, like USA and Brazil, the algorithm, the model, can reinforce 8:27 those representations or those categories. And this is a problem, per se, and a likely answer to that is to improve the databases. 8:41 But something that we usually try to advocate for is that some of the 8:49 uses of artificial intelligence can be discriminatory, per se, when we think about using artificial intelligence to incarcerate people. 9:01 So there are a lot of, uh, ways to mitigate algorithmic discrimination, and this can mean, uh, improving data sets. This can means, uh, 9:17 hiring more diverse teams to develop the systems and more, and, and hire more people, uh, committed to really, uh, audit, understand, and 9:33 plan for, uh, im- for possible harmful impacts, but also, in my point of view, and to a lot of Brazilian activists, even banning some kinds of technologies, like facial recognition in, in Brazil. 9:50 And if we could just focus on facial recognition then, uh, and you mentioned there in your answer the idea of incarceration being linked to that. 9:57 We've seen a lot of surveillance systems implemented across Brazil that use facial recognition technology. 10:04 I read last year that the state of Bahia celebrated catching their 1,000th fugitive using their facial recognition tech. How effective or ineffective have these systems been in Brazil? 10:15 I think the first answer, uh, to this question is that the efficiency of those systems are not fully understood by human rights defenders or researchers in Brazil because of a lack of transparency. 10:36 This is a first problem related to policies in Brazil. 10:41 And this case of the Bahia state, and it's my hometown, uh, so I'm very, uh, touched by that, uh, is that Bahia is implementing facial recognition since 2018 with a huge lack of transparency and 11:01 not, uh, clear procedures for procure, uh, those, uh, technologies and companies selling these software and this infrastructure to implement facial recognition. 11:17 And, uh, additional problem in a political sense is that Bahia is one of the most left-leaning, uh, states regarding voting in Brazil. 11:33 So the main center-left, uh, party in Brazil, Partido dos Trabalhadores, is governing Bahia for 20 years. Uh, and they are also very focused on carceral solutions for problems, but they are not efficient. 11:52 A recent study from the organization OpenOptical discovered that Bahia spent, uh, 600 millions of reais, or currency, to implement facial recognition in 70 cities in Bra- in Bahia. 12:12 And we don't have, uh, enough data on the efficiency, but we do have data that a lot of those cities are not, does not have, uh, 12:28 public sanitation services, for example. So this is a problem of priorities of public spending. And even in cities without, uh, wi- without numbers of violent crimes, 12:47 those technology, technologies are being implemented. 12:51 So this is a problem around efficiency and public spending, and it's likely that the amount of money being used by the state to implement those technologies, uh, is for two reasons. 13:10 One of them, it's a political instance of 13:17 faking that those technologies can reduce crime, and this is not true because the crimes in Bahia, violent crimes, did not, uh, drop it during those last five years. 13:32 And another problem is that there is a huge lobby from private companies, especially foreign private companies and their representatives in Brazil, to sell these technologies because, as I mentioned it, it's a huge amount of money being imple- uh, being used. 13:52 Only Bahia was 600 million reais. So other states are implementing facial recognition, like the most populated ones, like São Paulo and Rio de Janeiro. 14:05 And sadly, this is being implemented and used in political discuss by governors from center left to extreme right, uh, political leanings. 14:21 So this is, uh, how can I say, a national problem. Yeah, it seems to be that whether you're on the left or the right, politicians in Brazil seem really keen on using this technology. 14:32 I mean, not necessarily thinking about the potential risks or shortfalls, right? Yeah. Regarding that, that is, uh, a very interesting research, uh, made by a, a researcher called Paulo Vitor Melo. And during the 14:50 municipal, uh, elections in 2021, he analyzed the discourse of the mayors that were elected in 2021. 15:03 And 17 from the 26 mayors included discourses and ideas of improving, uh, security and improving and solving crime in their cities, mentioning, uh, facial recognition, 15:24 even when we don't have a history of evidence and data on how biometric surveillance in public space can solve crime. 15:40 This is not, uh, a data that we have, and probably we have the opposite. In the cities that organizations studied the crime rates, like Rio de Janeiro and Salvador, 15:56 the crime rates did not drop it. So facial recognition is being implemented and used by a racist police 16:06 having problems, individual problems like harmful approaches to citizens, to innocent citizens, and also eroding the public space in the sense that 16:22 distance biometric surveillance can harm democracy itself because it, uh, weakens the safe, the safety people can, uh, 16:39 can have in the public space around freedom of movements and also freedom of being a political actor in the public space, like demonstrations or 16:56 political associ- associations in the streets can be harmed by the presumption of surveillance in the public space. 17:05 And if we can just veer slightly away from this debate and look at something that's connected, uh, and which I'm curious about, because we're talking about these facial recognition systems and artificial intelligence being used by municipal and state governments, but how does all this fit within Brazil's data protection laws? 17:22 Because if I'm not mistaken, the way that these protection laws work in Brazil is that citizens need to give some form of consent for this data to be used, no? What's happening there? 17:31 Yeah, this is, um, a great question due to two reasons, in my opinion. The first one is that we do have a general data protection law, very influenced by the European one, and it's a good law in a lot of sense. Uh, 17:50 this, uh, le- uh, regulamentation is not fully implemented because the Brazilian National Data Authority is, uh, conceptualizing or defining some procedures, so they have a regulatory agenda for the next year to define things like the right to reveal ultimate decisions or- 18:16 How informed consent is really, uh, understood in Brazilian law. 18:24 But on the other hand, the general data protection law in Brazil does not cover the use of data by the states for public security matters. 18:41 So this law does not cover these kind of implementations. Currently, there is a discussion in the, in the Senate and the Chamber of Dep- of Deputies ar- around a 18:58 proposal of a bill around, uh, data protection espec- specifically for public security and criminal prosecution. So this does not cover that yet. 19:14 But another problem in Brazil is that the idea of informed consent that is, 19:22 uh, mentioned in the general data protection law is limited by the material needs of, of people. So I can give an example. 19:35 Uh, two years ago, if I'm not mistaken, a very important civil society organization in Brazil was, uh, distributing food for poor people in favelas and marginali- marginalized communities. 19:53 This is a great thing, obviously, but these civil society organizations got the support, the sponsorship, in a sense, of a Brazilian startup on digital identification. And this startup gave 20:11 as freely to the civil society organization, uh, a software to identify the bene- the beneficiaries of this program, and they asked them, 20:27 poor people, hungry poor people in f- food insecurity context, to give permission to be included in the database. So only if they give permission, they would have access to receive foods. 20:45 So informed consent is important, but in countries like Brazil, where we have millions of people in hunger, is limited by the choices people can have in some contexts. 21:02 So informed consent cannot be fully, uh, acknowledged when the person is in a 21:10 vulnerablized position or when, or when they are facing police officers, because the Brazilian policy forces are so violent that they killed 4,000 people in 2022. 21:28 If I'm not mistaken, this is four times the amount of people policy officer- officers killed in USA, and I think something like tenfold the numbers of the whole European Union. So we advocate for 21:46 more strong protections in Brazil, uh, compared to the European Union, uh, laws, because we do have more, uh, human rights breaches. 22:02 So we do ha- we need, in my opinion, to be more strict on defending human rights. Some really important arguments there, Tarcisio. 22:11 Uh, if we could just move on to artificial intelligence as a whole and the regulation of artificial intelligence in Brazil. 22:18 It seems like the Senate has taken the lead in that process and has been debating a number of proposals. Um, there's a bill proposed, for example, by the Senate president himself, Rodrigo Pacheco. 22:28 Could you tell us a little bit about that process and what proposals are on the table, and what does Brazil need to pay attention to when it's regulating artificial intelligence? Sure. 22:39 Uh, the discussion on artificial intelligence in Bra- on the regulation of artificial intelligence in Brazil got, uh, a lot of advancements in 2022. Rodrigo Pa- Pacheco, the president of the Senate, uh, 22:56 invited with the, with the Minister, uh, Cueva, a group of artificial intelligence experts to write a new bill proposal in 2022. 23:11 The first problem of this, uh, commission of experts is that this commission represented in a way the way that white supremacy works in Brazil. 23:24 Not a single person of these 18 experts are from a marginalized community, or none of them are from Black communities in Brazil. 23:37 And in Brazil, uh, Black people are the majority of population. So this was, uh, a scandal in a sense, but, uh, very likely in a very racist country like Brazil. 23:53 So civil society organization, uh, tried to push for more social participation in this commission, uh, that was writing, uh, a new bill proposal. 24:05 So by the end of 2022, this commission, uh, listened in a sense to civil society and a lot of civil society organization and researchers sent, uh-... 24:19 suggestions, and they finished, uh, reports on that proposal on how to regulate AI in Brazil. 24:28 And last May, in 2023, the Rodrigo Pacheco, the Sen- Senate president, submitted this proposal as a bill proposal to, to the Senate. 24:44 Uh, it's a advanced, uh, proposal. It's very similar, inspired by the, uh, risk-based approach from European Union. So there are some, 24:58 uh, possibilities and procedures very important, like the need for, for companies or developers to, uh, write algorithmic impact assessment reports when 25:17 the implementation is considered of right, high, high risk. 25:22 So if the implementation is considered of high risk, the developers have more obligations for transparency and auditing, and also submitting the, 25:39 the, the reports to a public database. 25:44 Besides that, the regulation, uh, the proposal, uh, included some points like the recognition of the possibility of indirect discrimination 26:00 and a, a whole chapter on defending human rights. 26:06 So this bill could be improved, but it was a huge, uh, advancement considering the, the former bill proposals that the private sector prefers, and they are lobbying for that, that did not enforce any, 26:25 any obligation for companies. However, 26:29 since the last, uh, semester, in September, if I'm not mistaken, there is another commission in the Senate comprised by this, by a, a group of senators led by senators like Marcos Pontes, a former Bolsonaro minister, and they are evaluating 26:51 those former bill proposals. And sadly, uh, the Senator Marcos Pontes, uh, offered a new suggestion, uh, for, for a bill removing 27:07 most of the human rights, uh, mechanism for defending human rights. Like, the whole chapter 27:16 around human rights was removed, and he prioritized protections for the private sector in the, in the name of quote, quotes, unquotes, "innovation and competition." 27:32 So right now, this commission asked for, uh, for, uh, for more time to finish their report, but we as civil society are pushing for more social participation in the discussion. 27:49 We had to pressure for some of us to participate in the public hearings, but the current, uh, 27:59 panorama is that probably this commission is going to propose just a series of recommendations without enforcement, uh, mechanisms. 28:12 So we as civil society organizations are trying to, uh, engage more people in tho- on those discussions because this could really, um, 28:28 move out for, from the advancements that we had during those, uh, two last years in discussions with other legislations like USA, European Union, the Chinese, et cetera. Thanks a lot for your insight, Tarcisio. 28:45 And I mean, clearly artificial intelligence will continue to be a very important topic this year in Brazil, and as these things develop, we'd love to have you back here to discuss that with us. Thanks very much. 28:55 Thanks very much, Ewan, for, for the space, and I'm available, so thank you. 29:00 And that was our interview with Tarcisio Silva, senior tech policy fellow at Mozilla and author of the book Algorithmic Racism: Artificial Intelligence and Discrimination in Digital Media. 29:11 [instrumental music] If you like Explaining Brazil, please give us a five-star rating wherever you get your podcasts. It takes only a second, and it will help us reach a wider audience. 29:22 Or better yet, you can subscribe to The Brazilian Report, the journalistic engine behind this podcast. We have a subscription-based business model, and your memberships fuel our journalism and keep us going and growing. 29:36 Thanks to our subscribers, we have been able to cover Brazil and Latin America extensively, and our work has won and been shortlisted for several international journalism awards. 29:48 Our newsletters, for example, won the Best Newsletter prize in the Americas from the World Association of Newspapers and News Publishers for a small or local newsroom. 29:59 In order to keep doing that work, we need your support. Go to brazilian.report/subscribe. I'm Gustavo Ribeiro. Thanks for listening. Explaining Brazil will be back next week. 30:13 [outro music]