[0:06]I'm I'm not used to speaking in person anymore. Um, this is, uh, become a rather rare event. Um, so thank you very much. I'm really honored to be here. Um, I really appreciate the invitation and, uh, let's get started. So, I'm going to talk about a little bit, uh, of experience that I've had, uh, over the last few years when I've been talking to a lot of policy people about technology and trying to go deeply into the relationship between technology and society. And so that's kind of gonna be the theme here. So, um, you're all familiar with deep neural networks. They're, um, their structure is nothing new, I think, I assume to everyone in the room, okay? Uh, the way that they're realized on today's computers, the algorithms are really relatively simple. Uh, I've worked in software for 40 years, I work on software systems where the complexity in the algorithms is vastly more than what you find in neural networks. That doesn't mean that it isn't challenging to develop this software, of course it is, but it's mostly about getting performance out of the machines that we have, and not so much about the complexity of the algorithms. So, um, one of the issues that comes up here is, um, the cost of doing this of course. There's a, uh, estimate, uh, by, uh, Struble et al at the University of, uh, uh, Massachusetts at Amherst that estimated that training one, um, natural language processing, uh, deep neural network costs 625,000 pounds of carbon emissions. Uh, to put that in context, that's approximately the same as the fully loaded airplane that I flew on to get here from California. Okay? It's, uh, a pretty substantial carbon emission involved. So, uh, despite the simplicity, relative simplicity of the algorithms, the amount of computation is of course enormous. And that's going to be one of the key themes that I'm going to talk about because, uh, in the context of policy, in regulating AI, one of the things that people tend to who are not experienced with the technology, tend to naively think is that if you understand what the algorithms are doing, you can explain any decision that the neural network makes. And, is that really true? Uh, if I explained to you the operations that are going on in the machine, does that constitute an explanation for what the neural network comes, comes up with? So, let's look a little more closely at what is an explanation. So you, an explanation is about answering the question, why? So you start with your input data, you give a sequence of logical deductions where each deduction conforms with rules of logic, and the sequence terminates with the conclusion.
[3:07]Now, this is a pretty classical description of computation. This is what computation is in the Turing Church sense. And obviously, these neural networks are implemented on a computer, and therefore, um, if you give this kind of, uh, structured, uh, description, you must have an explanation. But an explanation in terms of billions or trillions or quadrillions of arithmetic operations is not going to constitute an explanation in the sense that any human would interpret as a satisfactory explanation. So, uh, there's a very nice analogy for this that was, uh, introduced by John Searl, who's a philosopher at, uh, at Berkeley. You know, if you assume that everything that happens in the world is a consequence of the basic laws of physics, the interactions of protons and electrons, uh, then in principle, you should be able to explain the emergence of the Ukraine war in terms of electron, proton interactions. Um, but is that actually going to be a useful explanation of the emergence of the Ukraine war? Well, obviously not. And explaining the decisions made by a neural network in terms of the arithmetic operations performed in the computer is is is very much like that. It's not actually an explanation. So, what constitutes an explanation? Well, in terms of rational thought, what do we mean by a rational process? Well, a rational process is step-by-step reasoning using clearly explicable rules of logic, right? Um, but there's a problem, right? Which is that humans, it turns out, are actually not very good at this. So, um, Herb Simon, who may be the, maybe the only person who got both a Turing Award and a Nobel Prize. He got the Nobel Prize in Economics for basically, uh, debunking, uh, uh, a very fundamental principle in classical economics, uh, which is that, um, the agents in an economic system act, uh, to maximize some utility function. And, um, what Herb Simon pointed out was that these agents, as humans, uh, actually can't be doing that. because the utility functions that you have, uh, that all the economists were working with, uh, if you look at what it takes to actually maximize those utility functions, the algorithms are intractable. They're, uh, the complexity is sufficient that it's well beyond the human capability to do that maximization, to perform that maximization. Okay? Um, and humans can also handle only very limited amounts of data.
[6:06]Okay? So, in, in terms of regulation, there's a couple of silver bullets that get, um, bandied about all the time now. Algorithmic transparency.
[6:19]Okay? If we have, if we introduce regulations that demand algorithmic transparency, it's very predictable what's gonna happen. All right? What'll happen is that every time a neural network, uh, operated by a bank, issues a loan decision, they will provide for you a trace of all of the arithmetic operations that led to that decision. Um, you, as the customer, are free to download this multi-gigabyte file that gives you a trace of the, of the algo, of the arithmetic operations that produced the decision. That's algorithmic transparency. They have revealed to you the algorithm that led to the decision. Um, that obviously is not going to satisfy anybody, okay? But it will meet the requirements of the regulation if the regulation is poorly written. So, knowing the operations that are done by the computer does not actually help the human determine whether the decision is justified. This is, when I talk to policy people, this is a huge surprise to them because they say, well, surely the computer programmer knows, uh, how this decision came about.
[7:55]I've written these programs. I can tell you, I have no clue about how this decision came about. Okay? So, there's kind of a mismatch in the expectations of the regulation people and what the technology is actually doing. So, uh, the GDPR, the, the, uh, uh, European regulation that that was, uh, established, um, four years ago or so, that is responsible for all those annoying pop-up buttons that come up every time you go to a new website that ask you whether you're willing to accept the cookies or or have to go away, right? Well, it turns out the GDPR has in it a regulation that gives you a right to an explanation. Uh, most legal scholars seem to think that the way it's written in the GDPR, it's actually unenforcable, but there's a lot of efforts in the European Union right now to try to make an enforcable version of the right to an explanation.
[8:52]Um, so if you give an explanation in terms of one of these rational decisions, uh, if you don't have, if you don't bound the number of steps, it's not going to be an explanation for a human. Okay? Um, and so, as a consequence, well, how could we come up with an explanation? If you, if you, if you have regulations that ultimately are enforceable, that give you the right to an explanation, what are we, what are we going to come up with? Well, it turns out that psychologists have known for quite a long time that humans are very good at coming up with explanations, rational explanations for our human decisions, where psychologists can prove that the explanation is cannot have possibly led to the decision. Okay? Humans are very good explanation machines and will come up with post-facto rationalizations for their decisions. There's a beautiful story about this in the book by Daniel Kahneman, who is another Nobel Prize winner, uh, in his book is called Thinking Fast and Slow. And he reports on, uh, it's this is this is a now you're getting a third-hand report, me reporting on Daniel Kahneman reporting on another study, um, done, uh, by these authors, uh, Danziger, Levav, and, uh, Avnain Pesso, where they studied Israeli judges who were hearing parole cases, and found that there was a very high correlation between denying parole and the amount of time since the last food break. Okay? Now, if you were to go to any of these judges and ask them to explain the particular parole decision, they would have no trouble coming up with a rational explanation for you. They'll give you, you know, oh, well, this person had did this and did that and that, therefore, they're a high risk, and therefore, we will not give them parole. And their explanation is not going to have anything to do with the amount of time since the last food break, okay? And nevertheless, it's demonstrable that the time since the last food break had a significant impact on the decision. So, I have a prediction that as soon as we have regulations that require that have teeth that require, uh, an explanation, we're going to train AIs to provide a convincing explanation for any decision. So, the way this will work is you'll give the AI the data and the decision. And the AI's job will be to come up with a sequence of logical deductions of no more than a dozen steps that provides explanation that is convincing to a human for that decision. Okay? I can explain to you how to train such an AI. All right? First, you train a DNN so that given the case data and a decision, it synthesizes an explanation. All right? It's not going to be very good at that initially. Um, then you train a DNN so that given a decision and an explanation, it decides whether the explanation was generated by a machine or a human. And then you pit these two machines against one another. Well, this is the classic generative, uh, uh, adversarial network, uh, due to Ian Goodfellow, which has proved incredibly effective. It was the technique that was used to train AlphaGo, uh, to play the, um, uh, the Go game, okay? Uh, this is a very effective technique and I suspect that this could be made effective and you could create an explanation machine that given data and a decision will give, will provide one or more, probably very convincing explanations. If you think about it now, the regulation will have hugely backfired, right? Because these machines will get good enough at providing convincing explanations that it'll become very hard to overturn any decision that is made by the machine. Okay? So, that obviously is not the intent. Now, you could use such an explanation machine for perhaps a, a slightly less nefarious and and and unfortunate, uh, uh, outcome. You could, for example, have a human that is trying to make a decision between convict and acquit, have the explanation machine provide one or more explanations for conviction, have the same machine provide one or more explanations for acquittal, and then have the human decide which are the more convincing decisions. Okay? This might be appealing, it's quite risky, right? Because then, you know, is the decision here a scoring of the DNNs or is it a verdict on the case? And here, I think, this is not a technical question, ultimately. This becomes more of a sociological question, and it becomes a question of how do we want decisions made in our society, okay? And these are not technical questions, they're not questions that that we as computer scientists can address by ourselves. Okay? This is something where there's this whole movement that I've been involved in that started in in Vienna, uh, called Digital Humanism that is trying to recognize the fact that the technology that we're working on has such a huge impact on society, um, that we need to bring together very multi-disciplinary teams, people that include psychologists, sociologists, computer scientists, uh, legal experts, etcetera, and really elevate the level of discourse and discussion that we have amongst ourselves about how our technologies are working within our societal framework. Uh, because these kinds of uses of technology could be very transformative and have potentially enormous risks to society. Uh, there was this wonderful DARPA program, uh, explainable AI that actually, um, produced, uh, quite a lot of very useful insights about the complexity of the problem. You know, DARPA programs are by definition, every time there's a DARPA program when it's over, the problem has been solved by definition, right? And, so, uh, never mind that, this program actually did produce a lot of useful results, but the useful results are more about, gosh, this problem is really hard. And this is a particular chart from a, from a retrospective on that program that is looking at the psychology of, um, of the relationship, the, effectively the psychology of the trust relationship between humans and and and the AIs. And, and you can see a lot of things that are going to really be kind of difficult to deal with in a fundamental sense, like this test of satisfaction. Right? The satisfaction that a human has with an explanation is a psychological question, not a technical question, and yet it's going to have profound impact on how these machines interact with us in society. So, the fact that humans are very good at synthesizing explanations, um, well, okay, so how do humans really make decisions?
[17:21]Humans, not AIs, okay? Um, if rationality is not the whole story because our ability to do rationality in the Herb Simon sense is extremely limited. A few dozen steps if we're really good, okay? So how do how do we make decisions? Well, again, quoting Daniel Kahneman in his book Thinking Fast and Slow, uh, the fast thinking is what he called System 1. It's intuitive, quick, inexplicable decision making. It's the quick reaction that you have, oh, that person is evil, we should, we should convict them, right? Or something like that. And then there's the system two, which is the thinking slow, which is the rational decision making where you come up with a few dozen steps of logical deductions that lead you to the conclusion. Um, so both of these are involved in human decision making, and only when system two dominates does the true origin of the decision correspond to a rational explanation. When system one dominates, the actual explanation of the decision, if there really is one, is going to have to be in terms of millions or billions of neuron firings. That's what's going on in the system one decision making, and that's not going to constitute what any human would call a rational explanation. So, for system one, that's that maybe all we've got as an explanation is that millions of neurons fired. Now, my contention is that deep neural networks are much more like system one than system two. This is a treacherous thing to say and I've been misquoted in the press as saying that DNNs are are doing intuition. It's, that's not really what I'm saying, right? What I'm saying is that it's much more analogous to the the system one decision making than it is to the system two rational decision making that is going on. And as soon as we understand that, we're going to understand much better the limitations on providing explanations for what the neural networks are are trying to do. So, what is an algorithm? Well, I pointed out that rational decision making looks like classical Turing Church computation. You start with input data, you follow a sequence of steps where each step follows well-defined rules, and the sequence terminates with a conclusion. Okay? Um, so let's put explanations and algorithms side by side. And with only minor terminology differences, they're essentially identical, right? An explanation in this Herb Simon sense is an algorithm, but the most important observation is that it's a short algorithm. Moreover, that the rules that are used are not necessarily just, you know, simple rules of logic, but they're actually socially agreed upon, uh, rules. And it's the society is involved in the construction of these rules. What are acceptable rules, what are unacceptable rules, okay? Uh, but other than that, structurally, explanations and algorithms are identical, it's just that the the the length of the two is different. So, according to both, if you take the results of Simon and Kahneman and put them together, system one decisions are actually not rational processes. Um, so human decision making, since it almost always involves system one, um, really is almost always not rational thinking in the Herb Simon sense. Now, you could say, well, maybe system one decisions are, in fact, nevertheless algorithmic. There's there's kind of a whole bit of a cult in the scientific community these days that everything in the physical world is algorithmic. Okay? There's a term that people use sometimes called digital physics, that ultimately the physical world is an algorithmic process. Um, in the book that I published called Plato and the Nerd, I, I show that as a as a hypothesis, that hypothesis is untestable by experiment. Okay? So that means that if you accept the Carl Popper philosophy of science, it's not a scientific hypothesis. You can only take it on faith, all right? The idea that everything in the, in the physical world is algorithmic is not a testable hypothesis, is therefore not a scientific hypothesis, it's something you can only take on faith. There have been people who have been trying to give explanations of neural processes, human intuitive decision making, etcetera, in terms of neuron firings, uh, some of the best work done in this area is done by Jeff Lichtman at Harvard, who has done very thorough maps of interconnections of neurons. Um, uh, Lichtman coined the term Connectomics for the idea of studying neural systems, biological neural systems by constructing, uh, and analyzing the structure of the interconnection. And Lichtman, uh, did a very detailed work on this and came to the conclusion that the original goal of Connectomics is unachievable. That understanding fully the structure of a biological neural network will tell us nothing, uh, almost nothing about why or how it actually operates, okay? So it's an interesting negative conclusion from the original objective of that research. But the fact is that many machines are actually not usefully modeled by algorithms. You don't have to even go to biological neural systems to find machines that are that are much better described by processes that are not in fact fundamentally algorithmic. In fact, it may be that deep neural networks themselves are not fundamentally algorithmic. Even though they're implemented today on computers, maybe what the computers are actually doing is brute force simulations of processes that are actually not fundamentally Turing Church computations, okay? There's evidence of this in the subfield of of reservoir computing, which if you're not familiar with it is a really fascinating sort of, um, a little bit of a side project within the the machine learning community. Where you take all the intermediate layers of a DNN and you replace them with what is called a reservoir, which is a chunk of physics that implements a very rich set of random non-linear functions. So you can take for example, I mean, a bundle of carbon nanotubes, put a few thousand probes into it for inputs and a few thousand probes for outputs. And what you've got is a very rich set of non-linear functions. It can't be trained, okay? The the carbon nanotubes are all fixed, but you've got a very rich set. And then you just train one layer at the end. And they've shown in this reservoir computing that that can be extremely effective, that you can get, you can match the performance of a neural network where there's training going on within the intermediate layers, okay? Um, so this kind of suggests that in fact, we may be able to realize, um, uh, neural networks effectively by dropping in chunks that we call reservoirs, that are in fact just rich non-linear sets of non-linear functions, uh, that are not fundamentally algorithmic, are not Turing Church computations at the at the roots. Um, many different devices have been shown to function effectively as reservoirs. The term reservoir comes from some of the early work which used just buckets of water. Um, so this is a fascinating area and it suggests that it's possible that DNNs in fact are brute force algorithmic simulations of non-algorithmic processes. Um, Tanaka, uh, in this paper, he's been one of the leaders in this area, makes this rather provocative conjecture that in fact, a biological neural network may have large chunks of biology that are functioning as reservoirs, where there isn't much learning going on at all. But the whole purpose of this bundle of neurons is simply to provide random non-linear functions. And that just having those random non-linear functions, if you have a rich enough set, is enough to be able to provide an effective learning mechanism. Now, there's another approach that you could take, um, which is to say, well, no, let's, let's actually study the neural structure and see if we can learn something from it. And there's some very nice work that's been done, uh, in on perhaps the most completely studied, uh, biological entity, which is this worm called, uh, C Elegans. Okay, here's a little picture of the worm right up here. Um, so C Elegans has 302 neurons, out of a total of, um, I something on the order of 950, uh, uh, somatic cells. So roughly a third of the cells of this worm are neurons. Okay? And the whole cell structure of this worm has been mapped out extensively by biologists. They have a complete map of the neural interconnection. And people have actually started to understand that there are certain substructures within these 302 neurons that have very specific functions. You can, you can actually identify kind of algorithmic processes like, um, moving from doing this class of actions to moving to the next class of action, so the sequencing operation. has been able to be identified. And there's some very nice work that has shown that if you structure your neural networks after something like this, where you have some understanding of of the sequence of operations that you're trying to do, that you can, in fact, implement a, a machine learning algorithm with far fewer neurons, orders of magnitude fewer neurons than is realized by just a brute force representation. So, for example, um, in this work that was a collaboration between several institutions, um, they they showed that, uh, parallel parking could be trained with orders of magnitude fewer neurons than a brute force technique, by just taking into account that if you build in your structure, the the sequence of operations that have to happen, okay? And separate the neural network into an architecture where there are separate subsets of neurons that are involved in each step, then you could get, uh, a very effective training with far fewer neurons. So that kind of suggests that maybe there's a path towards actually architecting these neural networks in such a way that that will provide perhaps better explanations. But when you start to get up to the level of the cognitive functions of humans, uh, this starts to become a pretty soft science. Where it's gets to be at least for anything we can envision in the near term, um, we're going to be, we're going to be stuck with trying to understand these cognitive processes in very empirical ways. where the the standard way of studying, for example, what regions of the brain, uh, of the human brain, uh, are involved in certain cognitive processes is about studying the effect of brain lesions. And it's hard to do, you know, controlled studies here because you get brain lesions that you can study very much by accident and it's hard to, uh, to to program these these studies and so, uh, coming to conclusions about how the architecture actually affects our decision making and then architecting our neural networks to fashion after that, this is a really long-term agenda and is going to involve, uh, uh, a lot of collaboration, I think, between, uh, a lot of different disciplines, biologists, psychologists, and computer scientists. So, in conclusion, explaining DNN decisions is challenging, and we can't oversimplify it. If we oversimplify it, we're going to get regulations that are going to backfire. They will, they will be counterproductive. Um, DNNs may be more like intuitive than like rational thinking. And intuitive thinking may not be ultimately algorithmic. And that's perhaps the most difficult thing, I, I expect most of the people in this audience are computer scientists who have been brainwashed into thinking that every process is an algorithm. It's not. And we can have a long-term conversation about that. Um, but the universality of algorithms is a myth, and it's really, uh, it's a myth that is deeply held by many computer scientists, and it's wrong. Um, architected compositions of DNNs may provide some explainability, uh, but we've got a long road ahead of us in figuring out how to do that. Thank you very much.
