Erica Thompson’s new book, Escape from Model Land, offers a fascinating and important perspective on mathematical models as being not just models of the real world, or real processes or systems, but also “subjective versions of reality” that encode all sorts of assumptions and value judgements.
In this interview with Brian Tarran, editor of Real World Data Science, Thompson talks about the “social element of modelling” and how it manifests, how to counter the subjectivity of individual models with a diversity of models, and whether human-made models are held to the same standards of transparency that are expected of AI-“created” models.
Erica Thompson is a senior policy fellow in the ethics of modelling and simulation at the London School of Economics Data Science Institute.
1 Timestamps
- What led Erica to write the book, and why now? (2:27)
- Critiquing climate models (7:30)
- Exploring the “social element” of modelling (11:36)
- Countering subjectivity with a diversity of perspectives (20:11)
- AI models, human-made models, and questions of transparency (25:54)
- Why write a popular science book about these issues? (30:11)
- Will the UK Prime Minister’s “maths to 18” proposal help or hinder our Escape from Model Land? (34:01)
2 Quotes
“Putting things in a mathematical language does tend to make people think that it is truth from on high. And so my book, in some way, goes towards saying actually, these models, obviously we hope that they’re based on facts and they’re based on data that we gather, but they also do have this value judgement content as well” (1:51)
“It is arbitrary how we choose to model a situation. There are infinitely many different ways that you could choose to simplify reality - this huge, messy, complex thing in front of us, with physical laws that we don’t fully understand and things going on that we can only measure by proxy.” (12:09)
“The choice of assumption has a very direct result in the model output and in the information and advice that you’re giving to policymakers… [In climate models] maybe we have a cost of however many dollars per tonne of carbon dioxide for nuclear electricity or for renewables. But what kind of price would you put on behaviour change? How many dollars per tonne of CO2 avoided does it cost to change the behaviour of a population such that they use less energy? If you put it in at $2 per tonne of CO2, it would be heavily relied on [as a policy response]; if you put it in at $2,000 per tonne of CO2, it’ll never happen.” (16:42)
“There needs to be more frank discussion of values and value judgments, and politics and social assumptions within models. And I think we are starting to see that with the pandemic models, particularly because it’s been so high profile. [But] it’s really hard to unpick your own value judgments. It’s easier for somebody with a different perspective to come in and say, ‘Oh, actually, you know, you’ve assumed that. Why did you assume that?’ When we are embedded in a particular culture of modelling, it’s particularly hard to imagine that anything could possibly be done differently.” (27:20)
“I think some people maybe read the book and think, ‘Oh, this is just a sort of woke advertisement for diversity’. Well, it’s not; it’s a way of doing the maths better. The whole point is to do the maths better, make better forecasts, understand the future more effectively, and be able to make better decisions based on that information.” (33:41)
3 Transcript
This transcript has been produced using speech-to-text transcription software. It has been only lightly edited to correct mistranscriptions and remove repetitions.
Brian Tarran
Hello, and welcome to the very first instalment of the Real World Data Science interview series. I’m Brian Tarran, the editor of Real World Data Science, and I’m very pleased to be joined today by Erica Thompson, a senior policy fellow in ethics of modelling and simulation at the London School of Economics Data Science Institute, and the author of a fantastic new book - which I have a copy of here - Escape from Model Land, which is subtitled, How mathematical models can lead us astray and what we can do about it. So hello, Erica, thank you for joining us. I hope 2023 got off to a positive start for you.
Erica Thompson
Yes, it has so far.
Brian Tarran
Good, good. Because the book came out, is it just before Christmas or just after?
Erica Thompson
Yeah, just before Christmas. So I’ve had all sorts of things flooding in saying, Oh, I liked your book, or I hated this bit or no, it’s exciting.
Brian Tarran
Yeah, no, well, it’s, I have to say, I think, I thought it was a genuine– I finished reading it over, over Christmas. And I think it offers a genuinely fascinating and important perspective on mathematical models as being not just I guess, models of the real world, or, you know, real processes or systems, but subjective versions of reality, you know, encoding all sorts of assumptions and value judgments of the people who, who create the models. And I mean, I guess that shouldn’t really come as a surprise, right? But, but is it a point that is often lost in the discussion around models, particularly where decisions might be, like, informed or driven by model outputs?
Erica Thompson
I think it is something that’s easy to miss. I mean, especially because we’re sort of, maybe as mathematicians were used to living in model land and doing things which, which we see as being logical consequences of previous things. And then more generally, the public look to science and mathematics and statistics as being objective arbiters, perhaps, of how things are and how things ought to be. And so, so yes, that that kind of putting things in a mathematical language does tend to make people think that it is truth from on high. And so my book, in some way goes towards saying actually, these models, they are, obviously we hope that they’re based on facts, and they’re based on data that we gather, but they also do have this value, judgement content, as well. And so we need to think about what that is and how we deal with it, and how we sort of express it and how we understand it.
Brian Tarran
Right, yeah.
Erica Thompson
Especially where we’re using those models to inform decision making or public policy, then it becomes particularly important.
Brian Tarran
Yeah, yeah, no, and obviously, your book draws up quite a bit on the Covid-19 pandemic, and how models were used there. But I thought it was interesting, actually, that, you know, in reading the acknowledgments that you– that while the pandemic lent the topic, additional relevance, right, you actually started writing the book before that. So what led you to think now’s the time? What was the tipping point, if you like, of thinking, I want to write this book now?
Erica Thompson
Yeah, okay. Well, that’s an interesting question. I mean, because it builds on the last sort of 10 or 15 years of my work. So I started out doing a PhD in climate physics. And my background before that was maths and physics. And so I was doing a PhD on the physics of North Atlantic storms, looking at how they would change given climate change. And so, obviously, the first thing you do is a literature review. And I started looking at different models and what the what they were saying about what would happen to North Atlantic storms. And what I found there was that there were models saying that the storm tracks would go north, they’d go south, they get stronger, they get weaker, they’d, you know, anything you name it. And interesting, particularly, interestingly, was that they, they had relatively small uncertainty ranges. So they they didn’t agree within their own uncertainty ranges. And that made me think, well, we this isn’t telling me very much about North Atlantic storms. But it’s telling me a great deal about modelling and the way that we do modelling and perhaps we need to start thinking more about how these uncertainty ranges are calculated, what does it mean? What, how can we end up in a situation where we have this level of disagreement between models. And so since then, I’ve been looking at, you know, those kinds of concepts in different areas I’ve been looking at sort of insurance and finance and weather and climate and humanitarian forecasting as well. And, and so in all of those application areas, I found the same questions about uncertainty and how we make inferences from model output to be particularly interesting and how common problems may be solved in different ways as well. So it’s interesting to do the compare and contrast. And so, yeah, then I guess I, I’d been on all these sorts of bitty little projects and thought actually, I’d really like to bring this together into something more coherent, you know, to actually say, look, there’s a, there is a common theme here and we need to be putting it together and drawing conclusions. And we can, we can learn a lot from doing that. And we can share the best practice throughout the sector.
Brian Tarran
When you’re starting down this path of, I guess, looking into the, I guess, the ethics and process of modelling, did it, was there a lot of other work that you identify that you could kind of draw on a lot of other thinking around this area? Or was it kind of under studied, under researched sort of aspect of the literature?
Erica Thompson
I think it’s under studied, I mean, of course, everybody who does some modelling, you know, you, you do your modelling, and then somebody says, Okay, we need to put some error bars on the outputs, and you go back and, and think about how we’re going to put the error bars on the outputs. And probably, I would say, most people doing that realise that it’s much more difficult than they have the time to do or the ability within the scope of whatever project they’re doing. But the aerobars, the uncertainties always ended up being tacked on at the end, you do it after you’ve done all the modelling, there’s less incentive to do it. And there’s less resource to do it than to make the model itself better. And I think that’s a very common story, that people realise that they ought to be doing more, but they just don’t have the time the resource, the ability to go and do that. So then yes, there are there are people, and there are particular areas that I think have taken more time to investigate this. So in physical science hydrology, I’d say in particular, has a very well developed history of thinking about the uncertainties in models, maybe because, you know, they are constantly being challenged by events happening, which were not within the models, you know, you’ve got your flood forecast model, and then something happens, and it goes way beyond what you were expecting. And you you have to go back and say, What does this mean for our modelling process. And other areas have much less well developed considerations of uncertainty. And so that’s where I think actually, we could we could really benefit by sharing good practice across these different application areas, because people have looked in different ways. And, you know, with with different levels of statistical interest, you know, some areas go into the stats, much more, some areas are very philosophical about sort of the conceptual foundations of how we should think about models and how we should think about the range of outputs that we get from models. And so what I’m trying to do is bring those together a bit. Yeah.
Brian Tarran
Yeah, I think you certainly achieve that. It is really interesting, the different, the variety of examples that you present, and the ways you talk around these issues. I did want to focus in particular on climate models, though, because, you know, I was looking around your website, finding a bit more about about you, and I noticed on there that you talk about, you no longer fly to conferences, and that’s in order to kind of reduce your own ecological footprint. So I guess I wanted to ask, you know, when you set about writing a book, and it’s going to be a book that’s kind of critiquing models and the ways that they don’t often agree? Did you have like a nagging concern that, you know, the points you wanted to make about models in general, but climate models in particular, that that would kind of lend fodder to the kind of groups that might want to discredit climate models or downplay the risks? Or, indeed, the reality of climate change?
Erica Thompson
I mean, yes, I did have that worry, I still have that worry. And I, but I hope that my book is clear throughout that, you know, that models are not irrelevant, you know, the answer is not to throw them away. If you come to it from this sort of sceptical position, saying, you know, we need to think more carefully about how we make inferences from models, you could go all the way down the rabbit hole and say, Oh, they’re all terrible, let’s just throw them away. But I think that would be completely unjustified. We have good evidence, sort of from from one end of the spectrum of relatively simple linear models, which are incredibly, wildly successful and form the foundation of modern life and modern technology. And, you know, with that, as a basis, we hope that we can, you know, work from there to find the limit of the knowledge that we can get from these more complex models, which are looking at making predictive statements in more extrapolatory domains where the underlying conditions are changing, and we therefore have less ability to rely on what I call the quantitative route of escape from model land, by challenging with relevant past data. You know, we’re looking at extrapolatory conditions like climate change, or social and economic systems, and therefore, we think that the data that we have, while they may be useful and indicative, are not, we can’t just calibrate with respect to past data and expect that to be enough to warrant performance in the future. And so, so I think that sort of one answer is that we shouldn’t be throwing away the models completely because they are demonstrable useful, and the question is to quantify the limits of what we can say, rather than just get rid of them. And then maybe the slightly more nuanced answer is that actually, if we have less confidence in the models, and our uncertainty ranges are wider, then because in many of these application areas and climate change, in particular, the damage function is convex, you know, we are expecting that as we go further from today’s climate, the consequence will be not just linearly worse, but sort of increasingly worse. And therefore, if you have, if you’re considering, you know, just to have a sketch of a kind of cost benefit analysis on some sort of expected utility from taking action to mitigate carbon emissions, for example, if you have more uncertainty, then your range is wider. And so the, the lack of quantification of the top tail becomes dominant in in the expected utility of the outcome. And therefore, you should be choosing to mitigate more, not less, because of that uncertainty. So, you know, the sceptics, I suppose the climate sceptics would say, oh, there’s a possibility that climate change might not be as bad as we expect, and therefore we shouldn’t bother doing anything. But I would say actually, that argument should be turned on its head, if we have greater uncertainty, that should be a bigger motivating factor to reduce carbon emissions rather than the opposite.
Brian Tarran
Yeah, and I think you make that point quite clearly in the book. And the other point you make is that, I guess, building trust in models is about understanding their limitations. And the quote that I thought was really interesting was about acknowledging the social element of modelling. And I wonder, do you mind explaining what that social element is for people who are watching or listening? And how will that kind of manifests itself in models? Maybe you’ve got like a simple example that you might want to talk to? I don’t know.
Erica Thompson
Yeah, okay. So, I mean, the social element of models is, because it is arbitrary how we choose to model a situation, there are infinitely many different ways that you could choose to simplify reality, this, you know, huge, messy, complex thing in front of us with physical laws that we don’t fully understand and things going on that we can only measure, sort of by proxy, with, you know, with models themselves to make many of our measurements of the system. And so, so you might choose to simplify a system in one way to model it. And I might choose to simplify it in a different way. And you might choose one programming language, and I might choose another and they would implement functions in different ways. And so all of our choices change the way that the model will then look at the end of the day. Now, then you say, okay, but supposing I’m modelling you know, what will happen to a ball when I throw it up in the air? Surely, that’s not a, you know, that has no social content, does it? And I’d say basically, no, it doesn’t really have any social content. It has some social content insofar as you’re deciding that this is what we want to make a model of. But ultimately, you and I would probably come up with very similar models, regardless of our background or our perspective, or our interests, or even our education to a large extent. And so, so those relatively simple linear situations, which I refer to as interpolatory models don’t have very much social content. Now, the ones that I’m particularly interested in and that I talk about in the book are things that are extrapolatory, where we’re interested in situations where we are trying to predict into the future a system where we expect the underlying conditions to be non-stationary, to be changing. So climate change is one example. Social and economic systems would be another example. And when we’re modelling systems like that, we have to be much more careful because we could choose to model them in radically different ways. We could, if you want to model an economic system, you might choose to disaggregate with respect to the social class of different households. And I might choose to make a sort of bulk model of the whole system with a representative household. And you could imagine hundreds of different ways to do these sorts of things. So maybe you think about pandemic models and how you could simplify it into individuals or you could make an agent-based model with, you know, actual agents walking around and infecting each other. Or you could just write some differential equations for how the transfer happens. So you could do it in, again, in many different ways. And the choice of simplification is then much more important, and it will have much larger first order effects on the outputs, and then on the framing of the question, you know. So you decide to model it in a certain way, with a certain kind of mathematics, and that changes the way that you might think about intervening in the system. If you’re presenting your model to a policymaker with the intent of informing them about their policy options, you might, if you have a model which can represent the effects of say, closing schools, or universities on pandemic transmission, then then that becomes a policy option. If you have a model which can’t represent those kinds of interventions, then it’s not a policy option. And similarly, with climate change, one of the examples that I talk about in the book is integrated assessment models of energy and climate. And so these are models which consider the energy system out to say 2100, and they put a price on nuclear electricity and renewables and all the other things that go into the energy system and basically say, how can we achieve our carbon targets at the lowest cost? Now, if you put in, if you choose to put in a certain price for a certain technology or assumptions about how that technology will develop in future, then you get a particular answer. And you put emphasis on certain kinds of policy options. And so that has, the choice of assumption has a very direct result in the model output and in the information and advice that you’re giving to policymakers. And of course, you might choose to put in something like behaviour change. So maybe we have a cost of however many dollars per tonne of carbon dioxide, for nuclear electricity or for renewables. But how much, what kind of price would you put on behaviour change? How many dollars per tonne of CO2 avoided does it cost to change the behaviour of a population such that they use less energy? Well, that’s not really in the models. And if it was, it would, again, it would be first order because that would be you know, if you put it in at $2 per tonne of CO2, it would be heavily relied on, if you put it in at $2,000 per tonne of CO2, it’ll never happen. And where you choose to put that in between influences how it looks, and then that influences the pathway that’s projected, and it influences the advice that you give to policymakers.
Brian Tarran
Yeah. You mentioned the example of school closures and stuff when you’re talking about Covid-19. I actually thought that, that one really helped me understand, I guess, and made it clear to me was that, there’s often been that argument about whether the lockdown was the right thing to do given the other impacts, but you actually say that if different types of people were doing the modelling, maybe if it was school aged people, and I guess encoding the impact that that would have had on them, and how much they value say not being able to go out and see their friends, the impacts potentially on mental health and things like that, it does change, I guess, the calculation of what the right intervention is or the right response is.
Erica Thompson
Yeah, exactly. And I think I think we haven’t anywhere near bottomed out all of these impacts of the pandemic, you know, both the health impacts, and also, mental health and economic impacts will be rippling on for a very long time to come. So we, you know, we can’t even now retrospectively look back and say what was the right decision? It’s really not clear, depends how you value the different outputs, the different outcomes of a decision. And yes, we didn’t have economic models of what the impact of lockdown would be. And if we, if those had been available and developed the same kind of mathematical complexity and credibility as the models of infection and transmission we had in those early stages of the pandemic, which had essentially morbidity, mortality, and the, you know, the impact on the NHS, you know, number of hospital beds occupied. Those were the bottom lines, and there was nothing else. And so that was given as an input to the policymakers. Now, that’s, of course not everything that the policymakers rely on, they have to, their role is to weigh up everything else as well. But if we’d had models which contained more information about all of those other impacts, I think it’s quite plausible that we would have seen different kinds of decision making. And then there’s also the communication aspect, that these models were used to communicate and justify and persuade the public of the importance of the actions that were taken. And, you know, I think it’d be hard to disagree that the actions that were taken immediately where necessary, we certainly did need some kind of lockdown straight away. But then the question of exactly what you do thereafter is much more difficult.
Brian Tarran
Yeah, yeah. So in the book, you kind of make the point that it’s somewhat of a fool’s errand to try and make models objective, and they can’t ever really achieve this, you know, principle of scientific objectivity. But that we instead should look to counter subjectivity of individual models with a diversity of models that do encode these different perspectives, like we’ve just been talking about. I wonder, you know, how would you see this working in practice? Or how would you like to see this working in practice?
Erica Thompson
I mean, that’s a difficult question. So it’s, it’s nice to think that, you know, we have these models, and they are unavoidably subjective. Essentially, my view is that the model encapsulates the expert opinion of a particular expert, and it comes laden with their own perspectives, and biases and preconceptions, as well as their expertise and their education and their experience of a subject, you know, which shouldn’t be set aside. So if we are trying to understand a situation, then we want to get as many perspectives as possible. And so in theory, incorporating the widest possible diversity of different backgrounds into modelling and making a multiplicity of different models and trying to see the problem from these different perspectives will help us to understand it better. Now, then the statistician will jump in and say, Aha, can we, you know, can we in some way average those models or use some sort of statistical inference to take those models and put them together and come up with an even better answer? And I would, I would sort of counter that by saying that there’s no reason to believe that our, that a set of models generated as essentially just one set of opinions will be an independent and identically distributed sample from some distribution, underneath which will be an estimator of, of the truth, if that even exists. And so many of the formal statistical methods that we would quite like to apply to an ensemble of models, a large group of models put together aren’t really conceptually valid at all. I mean, that doesn’t stop people doing it. And maybe you get some interesting information from it, but you certainly can’t rely on it as an estimator. So there’s a sort of statistical problem there. And then the other question is your reference class. So, to what extent do you believe that these models are all equally valid or equally plausible? So that then brings the social question back to the forefront. Because then you say, you know, if I believe that, you know, somebody from Imperial College, say, who is the head of an institute for epidemiology, and has many, many years of experience making this kind of model, you know, is an expert and is qualified to create a model and for that to be recognised as a valid expert opinion, who else has got the credibility to do that? How do we define that? You know, what do you call a plausible model? So then it’s a question of the sort of scientific gatekeeping. What kind of qualifications do you expect from somebody or from an institution? What kind of expertise counts as being relevant and valid expertise? Does it have to be mathematical expertise? Can it be lived experience? Can it be, does the model have to be a mathematical model? What kinds of mathematics are appropriate for the situation? If we disagree about assumptions, does that mean that we can’t consider the the two sets of models in the same sort of class of plausible models? Or are we going to start pruning it by saying, I believe your assumptions, and I don’t believe your assumptions? And if so, who gets to do that? Who gets to decide what is plausible and what is not plausible? And what is allowed to enter into this set? Because as soon as we start pruning it, then we make the statistical inference more difficult. You know, if you want to say, if you want to start applying your methods that assume that the models are on, you know, that the models are independent, then you can’t start pruning because then that introduces huge dependencies on your own expert judgement. So it just becomes extremely difficult. And this is where all of then the social questions about expertise and credibility and sort of scientific gatekeeping and how we assign that credibility and trust, trust in science, you know, who has trust in which kinds of models? This is something, this is a theme that we see coming out of climate science and, you know, hopefully less now than maybe 10 years ago. But in the pandemic, of course, we’ve seen it coming right up again with questions about lockdowns, and about vaccination strategies, and all of that sort of thing. Trust in science is really important. And maybe one of my themes is that trust in science actually is first order in the modelling process itself. It’s not something that is sort of added on afterwards, I’m going to go away and make my model and then the question is whether or not you trust it. Actually, trust and expertise and credibility are in the modelling process directly.
Brian Tarran
Do you mind if we segue to talking about artificial intelligence models, or models made by artificial intelligence? Because that’s, I think that touches on a lot of the same issues, right? And I wanted to think about, well, first of all, you say that, obviously, artificial intelligence models made by AI, they’re not objective, even though there’s like, they’re kind of building the models, if you like, those AIs have still been trained by people, been coded by individuals and those personal judgments and assumptions and all that get embedded into the artificial intelligence. But I think, I guess my question for you is, we’re starting, I think, to have a very frank and public debate about AI ethics, and to demand transparency and explainability of things like automated decision making systems. But do you think we’re kind of, are we falling short of holding ourselves as people to the same standards of transparency and being clear about the choices and decisions we make? And also documenting that subjectivity when we’re preparing these sorts of models and these sorts of decision making systems for policymakers to use?
Erica Thompson
Yeah, so I mean, I suppose there are two questions there. And one’s about what we do and one’s about the AI. So for the humans, maybe, yes, I think that there needs to be more frank discussion of values and value judgments and politics and social assumptions within models. And I think we are starting to see that with the pandemic models, particularly because it’s been so high profile. I mean, remembering that actually, it’s really hard to unpick your own value judgments. And it’s easier for somebody with a different perspective to come in and say, Oh, actually, you know, you’ve assumed that, why did you assume that? And, you know, when we are embedded in a particular culture of modelling, it’s particularly hard to imagine that anything could possibly be done differently. And so I think that’s, again, where diversity is really important, because introducing those perspectives will help to challenge dominant strains of thinking which can end up in sort of accidentally, and not deliberately at all, in a form of groupthink. So that’s the humans and then the, with the AI, yes, you know, they inherit their value judgments from their creators. And so, for example, on the statistical side, one might think about the kinds of loss functions that are used to calibrate machine learning programmes, you know, how does the machine decide what is better and what is worse? You know, it is learning to model a situation, but there will be some kind of loss function in there, which it is minimising in order to decide what is the best model. And so, being explicit about that loss function, I think, actually is really interesting, you know, the fact that the model has got written down an explicit loss function which it is minimising means that we can then analyse that and think about what are the value judgments inherent in that choice of loss function, which is something that we don’t have when humans are calibrating a model and they’re twiddling a knob here and a knob there and saying, Oh, does it look realistic? Am I getting the right kind of behaviours, you know, does it match up with the map I have from observations or whatever. And so, having that I think we can then say, what are the implications of writing a loss function in this way? And what are the values that are implied? I mean, even just modelling itself, you know, like choosing to solve a problem with recourse to mathematical modelling is a value judgement and implies a certain kind of solution, doesn’t it? So if we say that we even can come to a decision, that the models input will be relevant and interesting and help us, that is a value judgement.
Brian Tarran
Yeah, and perhaps we could return to that point a bit later, because there was a line again that jumped out in the book about decision makers needing to maybe curb some over enthusiasm for mathematical solutions. You do talk about documenting value judgments as being kind of one of five principles that you set out for mathematical modellers to adopt to support responsible modelling. I think it’s fascinating to me that you’ve used the vehicle of a popular science textbook to speak to this community, and to sort of set out these principles rather than, say, a journal paper or conference presentation. So I wanted to ask, is there a particular strategy to that decision? I mean, I have my own theory on that, but maybe I’ll let you go first.
Erica Thompson
I’d love to hear your theory. I mean, I guess partly because I suppose I’m very interdisciplinary. And I’m, I’m hopping between these different areas, as we were discussing before, so I have sort of a climate science community and statistics and data science and other application areas in humanitarian forecasting, sort of hydrology, geophysics, weather and climate and all the rest of it. And actually, I find it really hard to get these thoughts published in journal form, because I suppose partly because it feels too general for any specific journal. And perhaps it feels too simplistic that, that the reviews I get tend to be either Oh, we’ve heard this all before it’s not new, or this is too radical and this isn’t an appropriate journal for it, you know, that sort of thing. And actually, I kind of got to the point of thinking, Well, you know, do people even read these journals anyway? Actually, maybe really what I need to be doing is trying to provoke a wider discussion about models. And I do tend to get a, you know, a really good response, when I speak at conferences or talk to people about these things. People go yes, yes, you know, this is really important. Actually, this is something I’ve really struggled with, we don’t know how to do it, the uncertainty is always just an add on at the end that doesn’t have enough time allocated for it. But I don’t have the resource to do it, I’m not able to grapple with these questions, because they’re so fundamental and so wide ranging, and it would be really helpful to have more of a sort of walkthrough of how people tackle these questions in different fields. And so, so I’ve been trying to do that. And I felt that the book was a good way to sort of spark the conversation and maybe also get it to some different audiences. So I’ve had people contacting me since the book came out saying, oh, you know, I’m working in, like, asset valuation for disputes between states, really random things, quite different. And they say, actually, your book really struck a chord, and we have difficulties with this in this particular area. And so I’m really hoping that, you know, the book will help me then to find, to bring together people working on these sorts of issues with common themes from really different application areas and try to make some headway on how we can actually go about practically changing modelling practice to make it to make it work better, and assess uncertainty better. So it’s not just– I think some people maybe read the book and think, Oh, this is just a sort of woke advertisement for diversity. Well, it’s not; it’s a way of doing the maths better. The whole point is to do the maths better, make better forecasts, understand the future more effectively, and be able to make better decisions based on that information.
Brian Tarran
Yep, well, that’s not too far away from my theory on why you did it. I thought it was that it’s a great way of getting– if you can get policymakers and the public to read this, right, you can get them to hold modellers to these principles, rather than having it just be something that you kind of talk about within the community and it doesn’t really go outside that, right? It’s a way of people, you know, the next pandemic or whatever it might be, the next time a model is the focus of a debate, the public are equipped to ask the right sort of questions about the process. Okay, I’ve got one more question to you because we’re running out of time. And it’s back to that over enthusiasm for mathematical solutions point. I thought was somewhat serendipitous to read about, read that quote, in the same week that the UK Prime Minister Rishi Sunak announced a plan for all pupils in England to study maths to the age of 18. So, I wanted to ask you what you make of that plan, first of all, but also, I think, more importantly, does a more mathematically minded populace, are they better equipped to understand the mathematical descriptions of the world and that they are incomplete descriptions? Or is there kind of maybe some other curriculum that we need to tack on to this maths to 18, in order that people are able to better differentiate between model land and the real world?
Erica Thompson
Yeah, so I mean, I’m not a fan of– I mean, I like the idea of people studying maths to 18. I think more maths is a good thing. I loved maths, I still love maths, I think if more people were more generally numerate then society would be better and life would be better. But if you haven’t enthused people about the value and the interest of maths by 16, forcing them to study it for another two years is absolutely not the answer. It will just put people off staying in school past 16. So I, you know, I think that we need better teaching of maths before 16, rather than forcing people to study maths post 16. And part of that is helping people to understand how mathematics is relevant to the real world that they live in and teaching them the kind of things that they will use in their adult life. You know, people, most people don’t use Pythagoras theorem, but most people do need to fill in a tax return, you know, these sorts of things. Understanding orders of magnitude, and the difference between millions and billions, would be incredibly helpful, wouldn’t it? So, yeah, I think there are more basic questions there that need to be answered before we go into the details of sort of complex maths. And then, so what was the next question?
Brian Tarran
It was about whether whether you think, the more mathematically equipped we are, does that make us better able to understand the limitations of models? Or do we need something else to kind of train us or encourage us to think about these two separate realities, model land and the real world? And I say realities in inverted commas.
Erica Thompson
Yeah, I mean, I think the general public has a good understanding that the model land and the real world are not the same. There is actually a healthy scepticism of models out there. And I think that’s probably a good thing. I give a couple of funny anecdotes in the book about that. So I mean, one was a, I think, a YouGov poll about people going to the moon and saying, you know, would you go to the moon if you could be guaranteed a safe return, blah, blah, blah, and like, a large percentage of those said, Well, no, I just don’t think you could give me a safe return, they reject the model land. And then there was another example about intelligence analysts being asked to sort of calibrate a probability language scale. So they say, like, likely, however many percent and very likely however many percent and unlikely however many percent. And so the study was looking at different ways of doing that. And one way was to accompany the word likely, or unlikely, or whatever, with a written number of what the probability it referred to was. So it would say, like, likely, I can’t remember the number, but sat it was, like 50 to 70%, written down in the question, and then the question was, what is the probability of an event, which is deemed to be likely brackets 50 to 70%? And what people write down was not 50 to 70%. You know, as a mathematician, that’s completely ridiculous. Because the answer was in the question, why wouldn’t you write that down? But of course, what you’re seeing there is the rejection of model land. Somebody has assessed it as 50-70%. The question is, do you believe it? Well, no, actually, you might write something like 40 to 80%, because you expect there to be, you know, the model to be generically overconfident. And so this is sort of what I mean, by curbing over enthusiasm for mathematical solutions is that, you know, we have to understand that the mathematical solutions are living in model land, and that we can, in order to get out of model land, we have to say, do we actually expect this result to refer to the real world? Or is it only saying what the next model run is going to tell us? And so the act of doing that is difficult, and it’s more difficult for mathematicians than for the general public because as mathematicians, we sort of are used to living within model land and noticing when the answer is in the question and then writing it down. And we’re not very good at saying, Well, what’s my subjective estimate of the probability of this model being inadequate in some way? That’s not something that you can necessarily do with respect to data and so it’s a tricky one. So in terms of the over enthusiasm, you know, it’s curbing over enthusiasm, not curbing enthusiasm, because as I said at the beginning, and I returned to a lot in the book, actually, mathematical models are incredibly valuable. And they contain a huge amount of information and insight that we’re, we would be fools to throw away. But we need to understand it, you know, in a more nuanced way and be clear about what it’s telling us and what it’s not telling us. And that answers in model land aren’t necessarily the answers that we need in reality, though they may be informative about them.
Erica Thompson
Well, Erica, thank you very much for your time today, for talking through the book, which is out now. Do you have some some links or information about where people can find out more about the book?
Erica Thompson
Yep, look on my on my website, ericathompson.co.uk. And it’s available through all the usual booksellers.
Brian Tarran
Excellent, excellent. Well, I wish you the best of luck with the book. As I say, I think it’s fantastic. And well, I hope we get to talk again, maybe a bit further down the road and see whether some of these principles and this ethical framework that you talk about for mathematical modelling, whether that kind of comes to fruition because I think we need to watch that closely. So, Erica, thank you.
Erica Thompson
Thank you very much. Thanks for having me.
- Copyright and licence
- © 2023 Royal Statistical Society
This work is licensed under a Creative Commons Attribution 4.0 (CC BY 4.0) International licence, except where otherwise noted.
- How to cite
- Tarran, Brian. 2023. “How to ‘Escape from Model Land’: an interview with Erica Thompson.” Real World Data Science, January 25, 2023. URL