A conversation with AI researcher and writer, Kate Crawford, author of Atlas of AI. We discuss her new book, emerging forms of collective dissent, collaborations with artists, as well as her ongoing work as a composer.
references
Metric Systems (Kate's band)
transcript (auto-generated, may have inconsistencies or errors.)
Roddy Hello, this is Informer. The show that reveals the latest ideas from artists, thinkers, and technologists, and former invites you behind the screen. To me, the people sketching hacking and imagining the next versions of our world. I'm Roddy, Schrock your host. And in each episode, I spotlight creative minds grappling with a changing world through art technology, or often both. And I hope you'll subscribe to this podcast at informerpodcast.com, where you can also find show notes, links, and more information on all of the artists and projects that we discuss. So on today's inaugural episode, it is an absolute pleasure to reconnect with Kate Crawford. I've known Kate for quite a few years. I've invited her to be a part of numerous panels and projects throughout that time. And I'm always captivated by her eloquence in laying out emerging challenges and our relationships to technology. Kate is a principal researcher at Microsoft research, the co-founder and director of research at the AI Now Institute at NYU, a visiting professor at the MIT center for civic media is senior fellow at the information law Institute at NYU and an associate professor in the journalism and media research center at the university of new south Wales. She recently published a book called Atlas of AI. It's. This book really opened my mind about the ways that this current adoption of machine learning by society at large is really just the latest moment in which humans have been blinded by science and allowed our tools to guide our ethics rather than the other way around. So I asked her about this and on whether or not we've passed the tipping point where whether it's too late or not for us to take collective action, to be able to adopt AI in a more equitable way for the public. Good. And not just for a few. I also asked her about the role of art and opening up the possibility for collective action and more about her recent collaboration with Trevor Paglen, a project that went viral. So it's the middle of the night here in New York when I'm talking to Kate, but I was already up and excited to catch up with her. It was the middle of the day for her and Sydney. And I began by asking her more about her book, what it is, what it was about and why she wrote it.
Kate So Atlas of AI has been a very long project in the making five years of actually running it and, and many more years of actually researching it. And for me, it really came from the perspective of having been a researcher of artificial intelligence for such a long time. And noticing just how dominant this idea is that somehow AI is this spectral immaterial technology that it sort of code in the cloud, that it has no real kind of connection to the earth or the people who make it, this idea of, you know, disembodied ones and zeros. And I think that has very serious social and political ramifications. Certainly it keeps us at arms length from how these technologies are truly made. So part of the motivation of writing Atlas of AI was to really understand the full supply chains of artificial intelligence and in that sense to move away from the abstract algorithmic space and to ground it quite literally in the sort of specific locations and institutions where people are designing these systems.
Kate So, you know, to do a project like that really meant moving away from perhaps the more traditional academic approaches of, you know, being in libraries and reading papers, but to actually put myself in the locations where artificial intelligence is constructed all the way back from the mineralogical layers to actually go to the mines where the components are being pulled out of the earth to construct everything from, you know, consumer devices, to data centers, to all of the parts of planetary computation, to going into the places of work, all of the sorts of hidden labor, all along the supply chain that sort of kept away from view and this, this sort of shiny view that we're given of automation. And then also to, to, to look at the ways in which data sets themselves are constructed to go into the the labs where they're being made to look at the archival practices of classification over hundreds of years, and to see how they are now re-emerging and sort of zombie forms inside machine learning.
Roddy I love the way you're describing that. And when I was reading your book one of the things that really struck me was just the I'm thinking topographically or almost texturally, just the way that you saw this world, that so many of us are not exposed to when you went to the houses of AI, which I just think is a brilliant description. And you sort of visited all of these locations from the mines to Silicon valley. And I particularly remember your description of sort of flying into the San Jose airport. And your perspective as times almost felt as though you were looking at this strange American topography, from the perspective of like the woman who fell to earth, almost like everything felt so alien with such strange rules and unexplained behaviors. And I just wonder, could you talk a little bit about how you felt as you're going through that process? Did you feel like you were visiting an alien planet?
Kate I love the way you put that runny. I mean, in that sense, I wonder if it has to do with the way which immigrants will always see with a different perspective. You know, I, you know, I moved to America over a decade ago, but I've, you know, I've always been in that sense of stranger in a strange land. And, and, and certainly, you know, the, the history of the way in which immigrants have about the United States has, has been very influential on me. And along the way that then I think, and study the world and almost go all the way back to the talk bill and the fact that you needed someone coming up from France to do his PhD to, to really look at the ways in which democracy, as it was being constructed in the U S was actually, you know, running the risk of creating new forms of tyranny. And certainly we've, we've had a lot of experience of that in recent years, but for me, I think part of that Dean naturalization that you're referring to is, is a very conscious as is the attempt to really pull away the curtain, to look at the sort of, you know, to look at all us sitting there, pulling the lever.
Roddy I feel like some of the things you revealed are so are so strange and, and, and, and disturbing in some ways. And in one point in your book, you talk about how you feel as though AI and sort of a move to this data driven world that, that we're in really shifts the idea of image and relationality to other human images at a move from image to infrastructure, I think is the way you put it in particular when you're describing the usage of mugshots as sort of a fodder for AI. And what do you think are some of the effects of this move? I guess I'm just asking, like, what does it mean to start viewing other people as infrastructure rather than people?
Kate Mm, look, you know, it's, it's a really important question. And certainly in the book I look at it in terms of the way in which we train machine learning systems, particularly in the field of computer vision and in computer vision, if you want to create a system that can, for example, tell the difference between a image of a cat and an image of a dog, you feed it thousands upon thousands, sometimes millions of images of cats and dogs, such that when it's presented with a new image, it can detect a certain pattern in that image to say, okay, this is more likely to be a cat than a dog, but certainly when you start to see the way in which this works with images of people, is where I think that some of the very kind of problematic underlying logics really of computer vision become very stark.
Kate And, you know, this is one of the things that I researched was to look at the early training sets that have been constructed, certainly in order to create facial recognition. For example, one of the most common training datasets, which is maintained by NIST, the national Institute of standards for the U S is, is mugshots. And it was mugshots of people who had been arrested multiple times over their lives. And it's a really disturbing training set it's it's images of people at the most vulnerable often, you know, people who are injured, people who are distressed and crying and, and people of course, who had absolutely no ability to refuse this, this image taken, you know, it's completely outside of the framework of consent. And we have no idea if these people were, you know, ultimately imprisoned it's, you know, completely, they're just completely decontextualized images that are just used as an infrastructure.
Kate They're just used to train systems and to test your algorithms, to see how they perform over time. And it really, it shocked me. I have to say it it's one of the most sort of profound realizations for me as a researcher to see the ways in which large scale training sets over many decades have just been used in this completely depersonalized way where any meaning or care that might be given to the image of an individual person or to the context behind a scene is presumed to be raised at the moment. It becomes part of this aggregate mass that, you know, that we call training data. So in that sense, you know, all data is just seen to be material that can be ingested to improve technical performance. And that is this core premise in the ideology of data extraction right across AI. And, and that to me is a, is a type of original sin.
Kate I think once you make that shift, once engineers no longer even have to look at the images, because in many ways, these are training sets, which are used to train machines. So it's, it's not necessarily assumed to be for human eyes in that sense, that minute we stopped caring about the sort of personal histories, the structural inequalities and all of the injustices that come with these collections of images. That is when I think we, we build systems on forms of injustice. So certainly that, that shift, I think from image and all of the care, or at least, you know, sense of, of multiplicity of meaning and an image gets a raised when it, when it becomes this kind of infrastructural mass. And, and that's something that is, you know, I think a really important feature of how machine learning works today and certainly where it fails us.
Roddy Yeah. I, I hear you. And I think you, in one point described that as being a shift from what previously were thought of as a human subjects, as a way of talking about participants and consent in, in research to what you described as data subjects, where everything is quote unquote, an agglomeration of data points without subjectivity or context, or even clearly defined rights. And I wonder, do you ever speculate on how that's impacting kind of personal relationships or societal relationships?
Kate Absolutely. And it's interesting, certainly one of the ways in which I look at this in Atlas of AI is in terms of this issue of classification, we are being classified all the time, obviously, according to the, sort of the logics of ad tech and capital, as well as policing in the military. But so many of the ways in which we are currently classified by AI systems, according to these profoundly narrow, and I think, you know, deeply concerning an outdated framework. So, you know, for example, many systems still will try to automatically classify people into two genders in a male and female, despite how completely preposterous that isn't this day and age many use ridiculously narrow categorizations of race, you know, having, you know, four or five racial categories. Sometimes they include other, you know, as the kind of move to some idea of that, they kind of capture all things.
Kate And then some have these, you know, profoundly worrying forms of assessing people's Morat morality really, or character or effectiveness as an employee or emotion. And here, you know, we can think of, certainly one of the, sort of the chapters in the book looks in detail just at the many failures of emotion recognition, why the very idea of emotion recognition is itself. I think deeply pseudoscientific, but all along the way, we are seeing machine learning systems engage in the act of what I call in the book, making up people, which is a reference of course, to it. And Ian hacking essay from several decades ago now, where he looks at the way in which classification systems by states have, have often sort of made up human categories as ways to try and track everything from, you know, productivity to forms of surveillance, but we're doing it again, certainly in these machine learning systems that are, you know, primarily opaque to us yet, nonetheless, we're being classified and categorized in these ways, thousands of times per day, if not more.
Kate So the question then is does that change how we see ourselves and see other people? And so in the Ian hacking argue that it did, because the minute you create these classifications and people are classified, they in many cases begin to enact those kinds of classifications. They are told that they have a particular type of illness in the, you know, diagnostic and statistical manual. So they will be treated in particular sorts of ways. And, or in other cases, people will be told if they have particular sorts of features, they will receive particular kinds of benefits. And again, you change the social structures around behavior very, very quickly. And this there's questions about how much people are either able to sort of ingest if you will, these kinds of forms of machines subjectivity, or if people are actively resisting these, these kinds of modes of subjectivity.
Kate And I think it differs really in kind, depending on how much of it you can actually see and even engage with directly, which is really just such a small amount of what's going on because so much of it is of course, internal to the logic of, of systems that we don't get to look into. But certainly your broader question is, does it begin to change how we see ourselves and others? I would say absolutely. Yes. The question of how that sort of fine grained way that you would track and understand that I think is something we're all still figuring out. Yeah,
Roddy It really made me wonder Kate, it really, it really made me just wonder half we passed the tipping point. I just guess, I wonder, like given from where you sit and the sort of broad perspective that you have on this whole economy of extraction, do you have a sense of, you know, kind of where we are in the spectrum of like being able to make real positive change as a society versus the genie just being, let out of the box and we'll just see what happens.
Kate I do think there's an enormous urgency in finding ways to very seriously rebalance how much power we give to both the sort of broader political economy of tech companies, but also the systems themselves. And that needs to be done in, in some cases in extremely strenuous ways, by very strict forms of regulation, not just, you know th the lighter forms of, of, you know, public pushback. I think it's time we really actually need strong regulation, but that said, you know, I think part of the problem that you're pointing to is actually deeper than that. It's, it's a sense of almost technological inevitability. And I think certainly one of the things that I think is deeply problematic is when we believe that, you know, we can't change anything that EV that certainly these, these tools and these systems and these forms of power are inevitable.
Kate And, you know, we simply have to accept them as they are. And, you know, in that sense, it's certainly one of the things that I sort of write about at length, which is, you know, how do we create politics of refusal? How do you oppose these narratives of technological inevitability that, you know, posit that if it can be done, it will be. So in that sense, you know, rather than just assuming that these are, you know, the systems of power as they are, and the sort of technical organs through which they will continue to centralize that power, we should be thinking about, well, how do we actually carve out spaces of refusal? How do we create zones where these tools and systems simply can't be used to, to in some ways, push against what Donna Haraway called, you know, the informatics of domination.
Kate And I certainly think we've got some really good signs just in the last couple of years. Some of them are very localized. We could look at the sort of local bands of facial recognition or protest against algorithmic grading or the various cities. And in some cases, countries that are moving against predictive policing, we could look to the fact that the EU just recently drafted the first ever omnibus regulation for AI. But we've got a long way to go in the sense that all of these moves either, you know, highly localized and they don't necessarily help people outside of those, you know, generally quite wealthy and technically literate cities. But it also indicates to me that we have a real task on our hands in terms of really designing what good safety rails will look like around these systems, what strong regulation will look like and what forms of public dissent are going to be most effective.
Roddy One of the things that, that gives me a lot of hope and optimism are when I see sort of creative responses and artists driven public dissent that can not only help raise awareness, but I think that's sometimes too glib of a way to, to describe it, but to actually deeply encourage people who may just not be aware of what's happening to better understand their relationship to these machines that are just becoming more and more embedded literally in, in our lives. And one of the things that you did was you worked with the artists, Trevor Paglen in creating a art project that anyone could use and would help them understand what kind of big data and what big data sets were perceiving each of us individually to be. And sometimes it was hilarious and sometimes it was shocking. And other times it was terrifying, I might say. And and that, that project really went, I guess it went viral. I heard people responding to that and kind of seeing that project really sort of wake them up to these kind of hidden systems. And I wonder if you could talk a little bit about your relationship to art and kind of creative practice as a means of creating the kind of public descent or at least the public understanding that, that you talk about in your book?
Kate Well, look for me, some of the most rewarding projects have been collaborations and visual investigations and creative projects working alongside artists who really approach these questions from different perspectives than the so-called classical academic approaches. And I've had the great privilege of working alongside artists like Trevor, but also bled on Jola. And Heather, do we have bog and to be inspired by so many of the artists that have certainly been part of the IBM community but also part of the wider New York community of artists who look at these sorts of technological concerns, I'm thinking of, you know, Luke [inaudible], I'm thinking of Jeff or I'm thinking of Tega brain and Sam Levine Ingrid Burrington that, you know, it's, it's it's a long and fantastic list of humans who are addressing these sorts of questions. And, and certainly the project with Trevor Paglen was many years in the making.
Kate And, you know, we, we call it excavating AI because it really felt like, you know, years in the trenches of digging through training datasets that are used to build AI systems looking at, in some cases, you know, training sets like image net, which have 14 million images, which in many cases, I just don't think many people have looked at the images at all. Which explains why there were so many horrific things in there. And in other cases, you know, looking at the, the systems behind them, you know, how did these, how do these trainings that's changed over time? And it was really through working with Leif rig who is incredibly talented programmer in Trevis studio that image net roulette was constructed. I think that's the project you're referring to. The didn't go really unexpectedly viral. And, and the reason to make something like image net relate is really to give people the ability to see into the logics of a training dataset, because you can certainly, and we have given talks about image net.
Kate We've shown pitches, we've written papers, but the ability to train an AI model purely on what what's called, the people category of image net, that's the category that has images of people and classifies them with labels that, you know, go anywhere from boy scout to kleptomaniac and to allow people to upload their images and to see how would they be classified? You know, what, what sorts of labels would be applied to different images of themselves and their friends? That's when you really get to see the deep politics of these systems, that's where you get to see why, you know, how, which young people are labeled differently to old people and black people labeled differently to white people. And you get to see this sort of racialized, gendered and able as to logics underneath these kinds of systems. And, you know, for us in, I can remember the first time we gave a talk, Trevor and life showing image, net roulette, you know, a few people for it, all that was interesting, you know, but there wasn't, there wasn't really any, any kind of moment where it sort of caught fire.
Kate It was, it was actually six months later where somebody tweeted it out and it just took off. It was one of those really strange moments where suddenly it went from just a few people, really engaging with it to a million people uploading images a day. We had a real problem when I had, in terms of maintaining that infrastructure. It was, it was just, it was crazy for, for a brief moment there, but the beauty of it was to allow people to see the systems, but at the same time, being sensitive to the fact that many of these labels are truly offensive and triggering and harmful in themselves. And so we also want it to be very careful about how long we left, left image that we let out in the wild for people to engage with. So we kept it as a sort of a brief moment.
Kate You can still see it occasionally in a gallery shows that that Trevor puts on, but it's, it's something that we think is as a public moment, it made its point. And certainly a few months after we did the image net roulette project, the makers of image net released a paper saying that they were going to quote unquote mitigate bias in image net by deleting over 600,000 images. And many of the categories that, that we critiqued the question is, is that enough? And does that actually resolve the fact that, you know, this system, which, you know, image net has been out there for over a decade and had informed so many production level AI systems that are around us every day, those logics have already sort of gone into enter the sort of water supply, if you will, that's, that's all around us. So how do we think about the afterlives of training data sets in that context? And that's certainly something that we're continuing to work on to this day. You know,
Roddy I have to say, like, I you know, one of the thoughts that I kept having while reading the book as well was just that it, you know, I'm, I'm in a very privileged position. Like I can, you know, call you up and have a conversation. If I've got a question about data, I can read this book, you know, and I'm, but that's so different than the vast majority of people on this planet. And certainly in the United States, it seems to me that one of the issues is really around how there can be collective action in pushing for positive change given how urgent this issue is just in the way that we live and what the future holds for all of us on this planet. And are we simply overwhelmed in the responsibilities of what citizenship means right now?
Kate I don't think so in the sense that you're absolutely right, that if we look at the ways in which, you know, late stage capitalism is designed to keep us at a remove from seeing how the sausage is made, if you will, from understanding the many forms of extraction that go into the way that consumer devices are constructed in the profoundly disturbing political logics, that can be driving large scale AI systems in both the private and public sectors. I think however people are aware when these systems are touching their lives and producing serious forms of harm. And in that sense, you know, I'm thinking here of the many kinds of activism that we've seen in local contexts against particular types of algorithmic and AI systems. I'm also thinking of what's happened since the pandemic, and it's been such a horrifying time for so many reasons, but certainly one of those reasons is to see the way, the types of political structures that I write about an Atlas of AI have only gained traction, how we've seen just an increased centralization of power.
Kate We've seen even the richest people become even richer. You know, I can remember a few years ago when I flew down Jola and I did anatomy of an AI system, and we looked at how much money Jeff Bezos makes in a day compared to a cobalt. Miner is actually producing the cobalt for his systems. And it's, you know, honestly, you know, the, it would have to work 700,000 years to earn the same amount that Bay's OSS made in a day. And that was, then that was back then he's of course made so much more money now. And we're looking at just such extremes of wealth and such extremes of inequality that it, you know, it is beholden on last, I think to say that certainly one of the great, I think responsibilities of citizenship is to do what we can to address this just preposterous asymmetry of wealth and power, certainly in the west, but we can look more broadly. So I think there is a, a different consciousness emerging around things like climate change around things like racial inequality around certainly the ways in which we have responsibilities to each other and to our ecologies, it's been brought into relief certainly by the last 18 months,
Roddy That is such a key point and something that I think we can all think about more. And I'm sure it's going to be very resonant with our listeners to change topics pretty drastically. One thing that I definitely didn't want to leave out of this conversation was to just learn more about whether or not you're still making music. So
Kate The most recent record that we released just before, just before the pandemic began is in an outfit called metric systems. We released an album called people in the dark, which is, has only become a more relevant title which you can see on band camp and, and your various other forms of hegemonic music networks. But in terms of making music, you know, it is one of the things I just wish I had more time for. And certainly in the last year and all of the uncertainty that's, that's come with it. My, my priority has been much more on, you know, being present for my, my people in my community and, and, and getting the book out there less than making music, but certainly it's lovely that you mentioned it because one of the things that for me really reconnects me and makes me feel more optimistic about, you know, where we are in the world is actually writing music. So, so getting back into analog synthesizer mode and actually making, building modular sense is something that I also love doing, which has become a lot easier now than it used to be. Because, you know, there are shops where you can buy components and actually, you know, sold to them together. It's it's very exciting. So in some ways, little things like that are always going to be part of my life and certainly something I'm looking forward to making more time.
Roddy Oh, wow. I'd love this image of you actually soldering modular sense together. I'm just thrilled to know that that's something that you're working on.
Kate You've got a background noise bands, right?
Roddy Yes. I that was sort of my early exposures to, you know, creative technology were somehow taking a weird, but totally amazing sort of sojourn to Japan and working in a noise band and really having a very naive but enjoyable perspective of what technology was like in the early two thousands. And having that, you know changed a lot over the years, but I still hold on to just as you're mentioning, just that belief that actually thinking creatively and artistically and utilizing the best promise of technology as kind of a creative tool is, is something that can really recharge your batteries. And I guess in my case, remind me that there's, there's a version of, of these tools that is, can still be used for, for beautiful outcomes. Yeah, I'm going to totally check out your music and besides music what are you working on now and sort of, what, what do you see as your next sort of project?
Kate Well, there's this quite a lot of exciting things that are in the works at the moment, including a new project with the Don Jola. So the you know, we've, we've, we've taken from anatomy of an AI system back in 2018. We're now creating a project that I have to say is even more massive and complex. And instead of just thinking about space, we've sort of added in the fourth dimension and we're looking at time as well just to make our lives particularly on. So I can't say too much about that, but it will be I will tell you all about it when it is, when it is finished in public, but it's been really exciting to be working together again. And I'm also doing a project, a multi-year project, looking at the sort of deeper foundations of machine learning, the conceptual foundations, as well as the sort of practical making the sort of the epistemological layer as well as the, the, the material layer, if you will.
Kate
So that is something that has been really just such an intellectually confronting, but also engaging project is to think about, you know, are other ways of creating these systems possible. And it was interesting. I was just recently reading Jeff hop's fantastic new book living in data, which I strongly recommend. And he quotes Jack Halberstam who has this wonderful turn of phrase about the fact dominant history teams with the remnants of alternative possibilities. And for me, trying to think about the alternative possibilities for planetary computation is certainly one of the things that's animating my work.Roddy Well, I think that's a great note to end on. I can't wait to see what comes out of that. And I'm so appreciative of the work that you do, Kate. I really am glad to just have a minute to, to catch up with you both on a personal level and as well as for this podcast, it's really great to just reconnect and I hope to catch up with you even more in person when you're back in New York.
Kate Likewise, it's been such a pleasure, and I really look forward to the times when we're no longer doing these things on zoom and we're actually physical spaces together once more. I think that will just be marvelous and here's to that.
Roddy So as we were ending, zoom alerted us to the fact that it had stopped recording as it does, as we're all aware. And I sort of side and said, oh, you know, zoom, it's a, it's the way that we live now. And Kate wondered aloud whether we might in the future find out that zoom has been data. Mining are millions, if not billions of recorded hours of conversation all over the world. And whether that could be publicly revealed at some point in the future. Of course I think that's probably in the back of all of our heads as we have gone through the last 16 months, but just to ponder the possibility and think about what that could mean. It's a lot to take in. And maybe that's the reason that none of us are thinking about that as explicitly is Kate Crawford is until next time. Thanks for joining me at Informer. And I hope that you'll join us again.