Abstract

More argues that scientists rather than historians now set the public narrative on climate change, including the pre-industrial baselines behind the 1.5 degree threshold. He presents a geodatabase of about 55,000 geolocated climate records for Eurasia, continuous from 1000 to 1820 and built over ten years by 25 scholars. He shows how AI helped with quality checking by restoring a missing sub-database and splitting reports into one record per phenomenon, then tests the records against tree rings and the North Atlantic Oscillation, finding correlations near 0.5 between 1000 and 1300. The discussion turns to plain language for several audiences, telling gradual change as a story, and extending the data to more regions.

Chapters

Transcript

AI-generated transcript. It was produced from the recording by speech recognition and cleaned up with AI. It has not been edited by the speaker and may contain errors. The captions on YouTube are AI-generated in the same way. Each timestamp links to that moment in the video.

[0:04] Hey, I hope everyone online can hear me and thank you for coming. I'm Alex More. I'm an associate professor at the University of Massachusetts and I'm also a group leader, which is I think a made up title at Harvard for the Initiative for the Science of the Human Past, where I've been associated for a long time. Most of the work you'll see today has come out of that initiative and a grant that I first was handed in 2015 with no hope of renewing. And I renewed twice for a total of a million dollars. We actually started doing this type of work with the Machine Learning and AI and the Initiative for the Science Center passed back in 2007. And then kind of just went really analog for a while. And recently I became a skeptical user of AI and eventually was convinced that it enabled us to really do some incredible stuff. So this is just probably 20% of my work,

[1:09] and this talk will really be about environmental history. And the problem that I'll be addressing is the fact that scientists are consistently setting the narrative for historical trends with natural records. So all the news you hear about climate change, all the news stories, new articles, CNN, New York Times, whenever there is a new study, a new discovery about climate change or about trends in, for example, storms and change over time, And generally speaking, those articles, that media is repeating information from science articles. Historians are largely absent from this narrative. And to me, that is a problem. Pre-industrial historical records are considered especially unreliable or

[2:10] statistically not comparable to natural records, meaning scientists just don't know what to do with us and our data, historical data. But on the other side, we have everybody constantly reminding us about the 1.5 degrees, which if you actually ask the average person, they don't know what 1.5 degrees Celsius means. It's the threshold above which we should not exceed temperatures because of climate change. It's based on a baseline that comes from the pre-modern, pre-industrial world. My world, the world that I work on. But we are absent from that debate. So the baseline for whatever is pre-industrial and normal in climate and food and health, the baseline is set by scientists who can't read the languages, haven't studied history. They are basing their conclusions on natural records that can't speak, can't feel, have never actually lived a lived life, that

[3:14] have left no records. it's much more interpretive than a historical record, in my opinion. And by the way, I say this as someone who has a foot in science much of the time because I actually sit in the Department of Science. And I've worked and I have a postdoc from a scientific institute, Climate Change Institute, University of Maine. So historians need to reclaim history and set the narrative for the public. this with large scale data sets. I realize not everyone is familiar or comfortable with working with them with this type of thing. And I'm going to try for the next 40 minutes to convince you that it's actually a lot of fun. This is my friend, Johan Rockström. If you are at all anybody, if you're alive in the climate space, you know who he is, is director of the Potsdam Institute, probably one of the most visible climate scientists or climate policy people out there.

[4:16] He was at Climate Week, which I chair in New York City. I chair a program every year. And he was one of the keynote speakers on the opening night, along with Michael Mann and a few others. And I would like to point to on the screen, and I know you guys probably can't see what I'm pointing at at home, but the graphs. his latest planetary boundaries are talk and these he has many many graphs like this and he's pointing you can see this is the shadow I love this photo because it's got the shadow of his finger pointing at it and that is time scientists are doing history all the time and we are not there. So one of the solutions is we have to talk to scientists and start showing the

[5:19] value of what history is. And it's not that they're mean, it's just that they don't know how to incorporate our data and we don't know how to give them something that they can incorporate and vice versa. They don't know how to give us something that we can incorporate in our... So this is really a talk about having a foot in both worlds And I'm afraid, and this is just my personal experience, that if we don't do this, we're going to be disciplined out of existence. At least that has been my experience since I got my PhD, but hey, I could be wrong. So one of the solutions, in my opinion, has been to create a geodatabase, and that is basically an Excel sheet. I know the word may be a little difficult, but it's basically an Excel sheet of climate records from historical sources that have been geolocated, have a latitude and longitude can be found on a map. This was part, again, of this project that the Arcadia Fund of

[6:21] London funded for about 10 years. the coverage of the geodatabase, which at this point has 55,000 records, is from actually the year one, because there is no year zero, to the year 1820. But it is really continuous from the year 1000 to 1820. And the reason why is that I really am a pre-modernist. I like manuscripts. I like pre-industrial stuff. I like to spend my time in archives. And that's really why I really tried to focus on that particular period. I wanted to use it also for my book. It's been produced by 25 scholars of all ages and trainings that cover the entire Eurasian continent, the two continents with the Mediterranean in the middle, but very strong in Northern Europe. More than 10 languages, and it is one searchable user-friendly database.

[7:26] This is not intended to be one of those databases have 15,000 files that you don't know how to use and put together and can never actually get anything out of it. This is user friendly. So we went from this. Any type of source. Including manuscripts, including archives, including notarial records and all languages, including also secondary source additions. and including the wills and testaments of heads of state. That's a quarter there for scale. I had to transcribe the whole thing. To this. And this is basically what the database looks like. And there are references to where you can find the source for each and every record. i want to make clear one thing because i've been giving talks like this for now

[8:28] about 10 years and when you show new technology although this is very friendly audiences have understood but when you show new technology to historians the historians think oh my god uh really cool i'm scared to death i don't know what you're doing and by the way aren't you reducing the depth of a historical document to a number or just a row and totally opposite the sources are being ignored history is being ignored this is a way to actually restore the centrality and respect for historical sources and lived experience of populations restore the respect for it give it the weight that's necessary to give it in setting the narrative for various parts of the pre-modern world, but in this particular case, the environmental history. So this is not to discount them and make them into mere numbers. Historical sources are in fact useful to scientists, and there are scientists out there that will follow you if you just give them what they need.

[9:31] so why are we doing this besides restoring historians to the and putting historians back in the in the driver's seat climate change is a slow disease uh and eyewitnesses remember catastrophes so what you most environmental history is about catastrophes little ice age the great famine the the Medieval Warm Period, the Roman optimum, the Krakatoa, whatever. It's always about outliers. But if we've learned anything, telling the story of climate change to the general public today is extremely difficult because it is a gradual story, and most people don't notice gradual change. And to set those baselines, to understand how much farther than normal we are, we need to understand that gradual change. And to do that, we need very large data sets. And we need to be able to test those data sets and understand that gradual

[10:36] change over time. We cannot rely on modern interpretations of abstract numbers coming out of an ice core or a tree ring only to set the narrative and understand the baseline of what's normal. That's really the question we're asking, what's normal? It's a fundamental thing. So, three parts of the talk. We're going to be quality checking. That's where AI really came into play. We're going to talk about how we go from reports, textual, to a map, something that the public can understand, that anybody can understand. And then testing the data against the natural record so that the next time a scientist comes along and says historical data cannot be used because it's not statistically significant, you can say geodatabase. so um we worked for 10 years with different uh people 25

[11:38] total more or less at any given time uh summers winter breaks over the course of semesters hiring people from all over the place to actually do this um it was really exciting it was maddening to organize and mistakes happen When you have a large project like this, mistakes happen. They are bound to happen. I'm going to focus on a very, very basic one. We found many others, much subtler ones, but I'm going to focus on a very basic one. This is a geodatabase in a dashboard. Oops, sorry. This is a geodatabase on a dashboard in a beta version two years ago. And if you are even a little bit quantitatively inclined, you will see that there is a problem in the data. Historical reports increase, as you would, as you hit print, they skyrocket. And then there is a gap. And the gap is in the 1600s and 1700s.

[12:43] And yes, there was a massive plague going around. and I'm one of those that really focus on those things and I thought, oh, maybe there is a plague that we didn't know about. No, actually not. I actually have written that article. It's not this one. What happened was that we just didn't include one of the sub-databases that our contributors had created. and after you've finalized and consolidated all the work of 25 different people and you've made some changes, adding that piece that's missing is going to be a problem. Not for AI. You just give it the missing piece and you fill it. And that's exactly what happened. The gap was human. AI found it. I mean, I found it too, but AI could fix it in a minute.

[13:44] I can't tell you how many months that would have taken. I mean, hundreds of human hours to quality check 15 ,000 records that were missing. And you're never really sure. But if you actually create a workflow with AI to give it the original pieces, the new pieces, do quality checks multiple I actually used multiple models just to make sure. You can actually fix entire databases that are so far unusable or flawed. That's a massive step forward. So this is a very simple example of how AI can actually be useful as a tool. It's a calculator you can talk to. I mean, it's essentially what we're talking about. More subtly, When you go through sources and you have, for example, a record that says, we heard thunder at dusk and hail followed in the evening, we

[14:48] woke up to a cattle die. That's three different things. Most of our contributors added one from that sentence. Added one to the database and moved on. Now, we have the original text, we have the references, we have everything. But what do you do when you figure out that some people, the people that were working on the low countries, for example, the contributors that were working on the low countries, added all three. They were very conscientious about adding every single phenomenon to the database. and the people who were working on late antiquity or the byzantine empire of the middle east added only um one or you know they were much less uh careful about uh adding um creating records from reports so multiple records from one historical witness how do you fix that

[15:48] you have to go back through the 55,000 records, find the ones that only have one record or one climate record from a report and add rows, add records, right? And do it again and again and again. And the more you do this, first of all, your eyes glaze over after an hour. You're looking at a spreadsheet that just makes mistakes. But the more you do this, the more insecure you become about mistakes. give it to AI, say, find every single one of these records that has more than one phenomenon. The text is in the database. And please add a row for each, every time. And just duplicate the row and say, you know, you add the hail, you add the cattle dying. You add a episodic. Done. Done in one week. possibly three months, four months, if you have a lot of money and a lot of people.

[16:55] I mean, I can't stress how revolutionary this is. Again, I started as a very skeptical user, and yet this is, it's quality check. It's a very simple thing. It's not really error prone. And again, you can version things, you can create versions of things to make sure you have an archived version, and then do quality checks again and again and again to make sure the model actually did it all. I would recommend the large context window, so the larger models do this better. So quality checking for large data sets, historical data sets, really you can't do without AI anymore. It's extremely helpful. Humans just don't have that capacity. Our RAM, not to use a computer term, but our RAM is limited. We can only do this for a limited amount of time before we make mistakes.

[17:57] And there are even unknown biases in just the way that people work. So if you have a collaborative environment with many historians working on a large project, as some of us want to do or have done, you're just going to hit biases. And we cannot be victims of our own biases and then fall to the criticism, oh, we can't use that because historical data is biased. Okay. Let's try to remove that. It's not like science is not biased. So once we have a solid database, what can we do with it that we couldn't do before? The concept of geocorrelating things. The main problem you're going to have when you're trying to create a large database of events, of things. I was talking with Clifford earlier, a colleague I just met. Thank you for an introduction, by the way, Professor Gouldy.

[18:58] And, you know, how do you remove duplicates? How do you know if two people in two villages reporting on a storm are actually reporting on two separate storms or the same thing? How do you create, essentially, how do you go from single reports, historical eyewitness accounts, to events, things that have happened that multiple people are reporting on? How do you remove that duplication? How do you handle that? All these questions. Can we correlate climate and health historical events happening at the same time, for example? So a storm and a lightning. Can we correlate a flood and a storm? So correlating within the same database, right? Independent events. And then can we correlate those records with natural records, which will be the last thing that we do? So what I did was I created boundaries around, assumed boundaries around

[19:59] climate events. So if you have a storm or a thunderstorm or hail, my distance that I assumed was 50 kilometers. If it's farther than 50 kilometers, it's a different storm. It usually turned out that the historical sources were talking about separate events in a shorter radius. So the historical sources themselves taught me how to separate the events. And I went with what a meteorologist would do, what my climate scientist friends would do. And the historical sources actually separated these things more accurately. And you could do this by geocorrelating them, basically creating geographic correlations between events, between reports. That is not something that we are taught how to do in grad school. I honestly know and I've spent a lot of time editing and managing a massive atlas of historical sources called the Mapping Past Societies Project at

[21:01] Harvard, which we started in 2007. Still out there. And I can tell you that there are very few people that know how to do a geocorrelation, how to correlate two dots on a map, two events on a map. Extremely difficult. Do it on a scale. Really, really difficult. AI, not that hard. actually ask it to do it fairly quickly. It will actually heuristically find the best statistical model to do that and then provide you with questions. Which one do you want to use to assess these advantages, assess these other advantages or disadvantages? You are the executive. You're making the executive decision of how you want to treat this data. But the important thing I would like to point out from this is that the historical sources themselves eventually determine the It's an overused word, but the granularity of the database. Because the sources themselves were clear enough, eloquent enough to separate these events.

[22:04] And then we started correlating these events to each other. So first we separate, you know, to make sure that one flood is different from another flood. And then we start correlating thunder and lightning, which you would expect to correlate because they usually happen together. But we want to make sure that we're really not seeing biases. We're not seeing flawed data. So we test this with things that shouldn't be correlated. We're asking questions to which we know the answer, right? Before asking questions to which we don't know the answer. to make sure that we're not seeing things, that we're not hallucinating reliability here. And guess what? Comets, eclipses, and weather... Sorry, comets and eclipses, that weather shouldn't be there. Comets and eclipses don't correlate with anything because they're not climate events. But they are reported as climate events because pre-modern minds thought that comets would affect climate.

[23:06] it um thunder and lightning are correlated three and a half times more often than anything else as you would expect hail and thunder same thing drought and heat same thing so the database is actually providing within its own reports um associations that you would expect that's pretty good okay so what can what can you do with this um i think i'm going too fast but any kind of weather uh health or economic map can be produced from this so you can go on the dashboard uh which will be available eventually and select uh which phenomena to show and you can create a map of that phenomenon. And you can actually select a year or span of years or a decade or whatever you'd like. Or you can even create a movie and see how things change over time.

[24:11] The place, unfortunately, is still only Eurasia. We only really had expertise for 10 or so languages. I have seven others Arabic and and other Middle Eastern languages but this is this is the limitation so far and it's really only continuous for 800 years before that it's kind of sparse and you can't really do statistics with that the best time to do this statistics to my great chagrin is between 1600 and 1800 which is not a period I'm particularly interested in but it is it is so examples i'm i'm hoping that you all are are champing at a bit to see examples this is an example of a map that will appear in my book of uh the climate during 1348 the second pandemic black death that's what it looks like lots of bad harvests lots of floods and price rising in the east because uh guess what people are dying and there's nobody

[25:14] to harvest And so food is expensive. This is actually something that I've tracked across the Mediterranean. How does price of food change over time? And how do different governments actually affect the price of food by subsidizing it or creating a provisioning system so that a city can continue to survive, even during a climate crisis or a pandemic? This is, again, 1347, and you can see the price in the size of a gold coin. It's kind of intuitive, I hope. The price is a lot smaller, lower, in places that have that kind of law, that kind of subsidy, that kind of approach. That's the point of the book that I'm about to publish. So the Byzantine Empire and Venice have the price controls and price fixing. All the others don't. that people survived and were fed better in those. And by the way, there are, if you want to cross into bioarchaeology, there is

[26:18] plenty of evidence to show that if you're actually well-fed, you'll survive disease much, much better than if you are not. And that's David et al. showed this archaeologically a few years ago. So this is actually important for public health as well, for survival. um again done it again for 1277 which was a uh of fairly uh famine written uh year and i'm i want to show these maps this these are so i created these maps by hand um because i'm a little control freak about this stuff but um but this is what claude did and i don't think i the right clifford told me i don't turn then turn the right libraries on but i'm gonna blow these three uh blow through these three just to show you what's possible um you can actually see a basically a weather map a weather change map right a climate change map over time

[27:20] and this in particular 1374 is one of the most well-documented years in the geodatabase and you can see that it is absolutely riddled with floods. Flooding is the single most reported climate event in our geodatabase. And this is actually true across centuries, across countries, across continents. Flooding is the thing that everybody reports on. And it is amazing that that's the case, because I'll show you how this so much more valuable um so question does this database see real climate or just human bias are we seeing what people just report which they choose to report or are we seeing a fairly semi-complete picture of of what's happening at any given year

[28:26] my answer to that was um well i wanted to basically test this against existing natural records right because the assumption with natural records as i said when i we started this conversation is that they represent what climate was that that's the baseline and because there natural records than scientists tell us. So it is a complete picture. And so, okay, if that is the assumption, because you guys have journals that have communications departments and you make it to the New York Times and we don't, but if that is the assumption, sorry, then let's see if our database matches the natural records, right? For that, I'm going to be slightly annoying and use statistics a little bit. If you're not familiar with the... I don't want to assume that anybody... I'm not mansplaining.

[29:27] I don't want to assume that anybody is familiar with basic statistical variables. But, you know, this correlation variable, lock, correlation by ranks, that's it. N is how many. Just so everybody is on the same page. A positive R means positively correlated, negative R inversely correlated. P is basically how much better than luck are you at doing this. This is all we need to know. I don't want to get into it. Otherwise, somebody is going to. Okay. So Chronicles, the written source of reporting burst. And if you actually just do a raw count of the data that we have in the report, you're going to get pretty insignificant results. You can't take the whole 800 years and try to correlate it to something. It's not going to work. You have to normalize that data, and then you have to actually take decades that are well documented to show that there

[30:30] are correlations. So the larger your context window, the lower your chances of correlating it to anything, if you will. The larger your time span, the lower the chance. So the best chances are in decades or years. And you, of course, have to bring your data to the same kind of sampling rate as a natural record. And the best thing you can do, and which AI is really good at, is scan this periodically in 25 or 50-year increments over 800 years. And as you do that, you'll find the really good decades that are really well documented, right? So we did that. I did that. And you can see that between one – I mean, our values are ridiculous. us. Between 1000 and 1300, the R value is almost 0.5. I mean, and it continues to be that way for the Middle Ages, basically, the late

[31:33] Middle Ages, early Renaissance. Very high correlation between tree rings and our geodatabase. So tree rings measure drought drought or or uh floods and so we measured uh the correlation between the geodatabase that that reported droughts and floods and the tree rings and you have a 0.5 correlation which is unheard of in historical sources it's just not something that happens um and it continues through uh it fades a little bit toward the the the end of our period We then did it by, so this was without geographic correlation, this is with geographic placement, and you can see that the correlation by region between the tree rings and,

[32:33] sorry, the North Atlantic Oscillation and our geodatabase is really good in the north, not so great in the south. North Atlantic Oscillation basically, let me show you this, is a phenomenon in the North Atlantic that basically shows how the low pressure over Iceland, the low pressure system, wobbles in contrast with the high pressure over the Azores. This wobbling determines our weather. the more low pressure system, the more prominent the low pressure system over Iceland is, the more flooding and rain you get. The high pressure from the Azores is, the more dry weather you get. The oscillation is actually calculated with a ratio of the two pressures. That's it.

[33:33] It's very simple. and it was discovered not very long ago, about 10 years ago. So what I did was, okay, let's see how this natural record based on ice cores and tree rings, et cetera, correlates with our geodatabase. And you see that there is a high correlation with Northern Europe and a very inverse correlation with Southern Europe. And if you look at a map of pressure over Europe, you will see that I got a really simplified one, you will see that there is a line in between the two, and that's the Alps. And that's exactly the boundary that this correlation shows. This is historical data that's matching a natural record. And it's saying exactly what the natural record, the natural map of the Northern Latin constellation is showing. Exactly the same thing. But I know that the correlations are kind of hard to visualize.

[34:34] I put it on a map, on a graph. So this is, it basically is a map of Europe, that you don't have the borders, but, you know, you got Rome, and the high correlation, high green, is the north, and the south is here. And you can see how it breaks at, what is that? Florence. So Florence has an inverse correlation with North Atlantic Oscillation. Northern Europe, Germany, Flanders has a high correlation. And the Alps are the border. So it's a visual version of this. Same thing. So both validations, tree rings and a North Atlantic Oscillation, are significant. Each is strongest on either side of 1600. That's an interesting finding. I'm not quite sure what it means yet. I'm just going to tell you that.

[35:38] But the p-values are pretty good. It basically means this is random. And p-values, you create a thousand different fake versions of the same thing to make sure that you're not seeing hallucinations here. And that's the test. and it's fairly fairly good so why all of this because so why why ai in all of this too why the big you know we you hear a lot of people talk about big data and even in history and i've heard it for as long as i've been a historian which is in my entire adult life and the real thing about The big secret about big data is that you only really see outliers. The outliers are what? Big catastrophes is what I said before. Nobody writes a book about something slowly growing because it's not detectable.

[36:42] And what I've just shown you is essentially indirect evidence that eyewitnesses put in their sources and showing that gradual change, matching natural records. So we actually just proved that historical records can take the pulse of the environment, even when the writers themselves, the eyewitnesses themselves, didn't know they were doing that. Because this only appears when you have large enough data sets that can be statistically significant. significant. And so, you know, a person in Munich telling you that there's a flood doesn't know what's that another person in Paris is writing about a flood too. These two people aren't related, but they're telling you the same story. And only when you bring this up, these stories together on large scale and detect

[37:44] them statistically and then match them and validate them, you can, in fact, say historical sources are taking the pulse of our environment, and you have to take them into account. This is much more telling to me than, and I say this partially as a scientist, it's much more telling to me, much more eloquent to me than a line on a graph telling me that storminess is increasing or decreasing based on what an ice core holds. And I use ice cores. I'm just saying this is more fun. You can zoom in on each of these points and find the eyewitness account. You can read it in the translation and the original language. You can find the page number. You can put it in your article. You can do whatever you want with it. And, of course, at this point, I think you're asking, where can I get the data? and the answer is it'll be published with the

[38:45] book which is coming out in February so I know this was more methodological than most people would like but I think this is why we're here and I hope this was useful to you and I'd be happy to take any questions Oh. Yeah. Yeah. And also, you know, this is a data set that has two audiences. So for the purpose of climate science, I know that the scientists demand that these climate records, these good kind of detail, questions about whether they take human recording seriously. but you've also kept the human dimension of governance and welfare

[39:47] in focus through these accounts of floods and price fixing so can you just say a little about what it's like to work with a project that has two audiences okay so the question for those who are online is how do you work with a source or source data set has two audiences. One is climate scientists that can use it to validate or enrich their own. There's only really four or five climate scientists that have used it, so they all work with me, because I wouldn't give anybody else a doubt until now. And the other one is, the other part of the audience is actually what I wrote a book about, which is essentially a book about what should governments do to respond to climate crises and health crises? And, you know, this is why the data exists. It's to tell that story.

[40:48] And the answer to that is that, like in any interdisciplinary project, I find it enormously exciting to have two audiences or three or a hundred Because it humbles you into using words and concepts and sentences that are understandable by anybody. I can't stress enough how often we lose the opportunity to make friends or have an audience at all or make it into the media. I ran a grant-funded program on science communication and just communication to the media. How frustrating it is when we use terminology, words, sentences, concepts that nobody can understand.

[41:52] I mean, the one that I really find disturbing is 1.5 degrees. You know, it's all over the news. I ask this to all my students. I ask to even colleagues, do you know what it is? they don't know what it is. And we're basing entire policies on it. Nobody knows what it is. It's an average of an average of an average of an average over time, over geographies, over... And it has absolutely no meaning whatsoever because if you actually go where people live, where animals are, you will see that the degrees are five degrees already beyond the threshold. So why are we talking about 1.5 degrees? It's so minimal. Such a failure of communication. 1.5 degrees, you know, What is the interaction of the average person with degrees? A thermostat. 1.5 degrees is going to make a huge difference, is it, in your life? Yeah? So that, I think, is the most fundamental answer, is

[42:53] it makes you a better communicator. And it makes you a lot more friends, because people understand you when you speak, and you're not trying to say my data is better than yours they're just saying let's see what we can do together and so i learned how to speak to scientists in a way that they could understand me it's not the 14th century it's 1300s um for example for example um and you know they they taught me things and they you can see them checking themselves constantly and it's i also find it slightly silly that as teachers, we should know how to do this really well. But we somehow, you know, this is one of the few places where I feel comfortable enough using a little bit more jargon. But, you know, we should be able to use this, do this more often. And if we did, I think there would be more interest, at least, in what we're doing.

[43:54] I don't think that it is beneath us to be more understandable. Yeah. Thank you so much for this. I think this is kind of a follow-up for the question. I really like your description of big data. The small drips of history need to be more legible, as well as the changes. And I think that really complicates the kind of intro methods course that we do in the community change, all actually slow change but because you bring up like the 1.5 degree i i think you started the talk by saying that if people don't understand gradual change very well right so in your own kind of setting the stakes for the project like people like an inflection point people people understand the pivot point or the change after the continuity right so how are you it's actually so explain kind of like the the stakes here are the same but we're focusing on the small drifts yeah but how do we like you humanize the story but for people who

[44:58] aren't going to go and click through and look at you know the flood and maples thank you so uh so the question is for those who are following um how do you basically how do you tell the story of gradual change in a way that that that people will actually be interested because it's it's the same question of why you know once somebody asked me, why does the one year of NASA funding, why can that fund NOAA, National Oceanic and Atmospheric Administration, we actually rely on for weather and storm preparedness, et cetera. One year of NASA, which is mostly just about space, can fund NOAA forever at current funding level. And the answer I got from a chief scientist of NASA was because rockets are cool. and and so how do you make this cool right and thank you that's my next book um it's called baselines and basically the answer is you gotta tell a story

[46:02] i think we are stuck in analytical mode and what this allows you to do is to merge many lived experiences into one lived experience that tells a bigger story about what it's like what was what was it like to have an intact ecosystem compared to now and how did that progression unfold one great book um that i think about for how to structure this and it's very unusual for me to actually bring up fiction because i don't read it uh it's over story um tracking uh of three species or one tree over several generations. How does that affect a life? How does that affect the rhythm of life in ways that we're no longer detecting, but

[47:06] that humans can definitely relate to? Right? And bring it all the way. Bring it all the way to 2026, if you can. Looking forward, not backwards, not a presentist story, but explain the present story from the I think that's the only way. As I said, I've run lots of conferences many times. And the only talks that people remember and are memorable are ones that tell you a story. And so that's my answer. I hope that I will have the license to do that for a group project. Yeah, go ahead. Two questions, one more technical and one more broad. is that you had a slide on how it rectified the errors on the 17th century data, and then I think, you know, you showed that the machine was more accurate, and then

[48:07] you had a phrase there that said no error. So just the painful question is, did you cross-check every single output that the machine did by, did a human do that? No, another model did that. Okay. It's impossible. A human will make mistakes. Sure. Okay. It's that simple. A human will make the mistakes. I will make... errors done by another. Yes, I triple checked that with three different models, three different tools completely, just to make sure that it wasn't just one, one model one just Claude creating. So I wanted to make sure that that compared to the original, there wasn't an artifact. And I imagined I assumed and I'm welcome feedback and criticism. I assumed that other ChatGPT would be able to detect

[49:07] artifacts between the archival copy that was flawed by human error and the fixed copy that was created by or completed by AI. I did not find errors. I do think that certain models are better than others. I'm not an expert at AI infrastructure, although I'm trying to learn better how to use it. So there could be errors. The drop question was, so I just love the project, I love the work you're doing, and I love that the final part about the conciliates, right, that you use objective human records to validate the natural records. I think that's great. I think it shows that, you know, it does validation of these messy human subjective forces. On the other hand, I feel also that about the scientists, I might say something

[50:11] like, look at my example, like, I told you so. Like, why did he spend all this time doing that when Matthew just used tree rings? finding so far, did you find any kind of discrepancies, like anything that the human records kind of show us that the natural records don't? Definitely, yeah, for sure. Yeah, so the question is whether the written records are showing things that a natural record is not. Yes. And the answer is yes. And I showed it before where I showed you that there is a correlation between tree rings and written records up until 1600s and then something happens. I'm going to have a conversation with my friend Ed Cook about it. I don't know. He's the guy that created the Old World Drought Atlas. One of the most lovely scientists. It's a good question. I think one of the things, and this is, I think, a major contribution. Rather than a flaw, it's a major contribution that we can make because there are, again, assumptions about these large data sets

[51:14] But a tree ring atlas basically has taken little pencil-sized samples of trees from all over the Eurasian basin. In fact, he's done it for Asia and also North America and South America, and then measured them and put them in a database. The representation of those trees is not complete. It's not all the trees. so we make assumptions that that is a complete and accurate representation of what we think drought or floods were in these various areas but only because there is nothing else to tell us otherwise and by the way there are plenty of discrepancies between natural records themselves uh which people kind of gloss over or curve fit or do whatever they want in order to just yeah well we don't know what's going on there let me just tweak that model a little bit um and that happens i kind of like

[52:18] to trust historical sources but i also know the limits of it of them uh i just say i just have a hard time thinking that there are sources out there I'm not using to tell a story. That's just that's been the bane of my existence and why I I I do these crazy things. So that's I think the approach I take. Yeah, go ahead. You didn't use any AI, the initial process going into the document. No, no. But given the documents were digitized, that's something that at least in the more recent, like in the last several hundred years, that would do well. Yes. So the question is, did we use AI to actually come up with the original database? And the answer is no. And would we consider using AI to enrich and create a larger yet database of the same thing? And the answer is yes, we were just talking about it. and would I use OCR for it?

[53:21] I've started using OCR to look at and digitize text in 2006 when I was a graduate student at Harvard. And we were in, those of you who have been, we were in the medieval studies room up on the fourth floor and we had a massive scanner and we scanned these super old books to then use a Soviet software called Abbey, also the name of my dog, that you would train. It was the beginning of machine learning, right? To train on different fonts to do precisely what you're describing. I'm not allergic to it. I just... It wasn't something I trusted yet. And I think AI has just caught up to the point where it can't do it really accurately. And Lou and I have... digitization process is still painful. There is so much digitized that we

[54:23] would reach a statistically significant critical mass very quickly. Even the archival stuff that I showed you that I use is all digitized on a super clunky Italian website. It's there. So this, for example, is the record of the first quarantine in history. that's actually yeah I think there are tools you can train to scrape it they the clunkiness of the website might actually be a feature so far that as they don't bar bots from scraping it right so you could theoretically get the stuff. I think you have to manipulate these images a little bit. Even I, having all of the Venetian archives from 1272 to 1500 in my pocket at any given time, in order to read that, I have to

[55:26] play in Photoshop a little bit to increase contrast and stuff. So I know it looks like chicken scratch, but you get used to it. Yeah. Are there plans to add more regions or what are the barriers to that? Money, time. Yeah. Oh, sorry. Do you want me to repeat the question? So the question is, are there plans to add more regions to the geodatabase and what are the limitations to that? And the answer is really just time funding, you know, in order not to be disturbed. But it will not require 25 people anymore. It will require three or four colleagues who are – and I agree, by the way, that the Spanish specialist, Spanish history specialist should be doing. So I'm a premodernist that really specializes in

[56:31] the Mediterranean. So my languages are Greek, Latin, Italian, and all the Italian, some Croatian, French, German. But really, you know, Greek and Latin and derivatives. I would never claim or supervise work on Arabic. So I think that that's one limitation. You have to find people who can do that. And are interested in doing that. Even if it's translated, I think it's important. So that would be one limitation. But there's so much digitized, as was just said, that even just starting from the digital, what's already there, it would be an amazing step forward. I think that if I had a quiet semester, I could blow this up. at this point to 2 million in two months, 2 million records in two months easily,

[57:32] very easily. It's just a question of, yeah, yeah, go ahead. So great. And I'm so happy to be here. This is really exciting. I'm always thinking about sort of data looping, data aggregation, and sort of larger data systems. And for instance, I have a lot of the same kind of data, but for North China, right? And it wouldn't be, and we should talk. And it wouldn't be appropriate for either of us to sort of merge our databases into one another. But I'm wondering whether you envision sort of building toward an information. Absolutely. Yeah, absolutely. All together kind of stuff in this domain in one sort of kind of searchable. Absolutely. Yeah, so the question is, there are other people of, I don't know the size, but I'm assuming sizable databases of similar data. Would you be interested in centralizing all of this in a place that people can

[58:36] use where it's searchable? So Mapping Past Societies, the atlas that we have that maps all of this stuff, is one such platform. We also have a data availability page where you can just download everything that we've ever produced. So it's all there. It's all open access. Arcadia insisted on it, the funder. So it's all open access. All our publications are open access. I mean, if you ask me, dream job, I would love to run a center that does precisely what you're saying. and then uses AI as a heuristic tool to see patterns that we're not seeing that can actually lead to policy decisions and even economic decisions, if you will. That would be fantastic. I would love to do that. I think it would be a great service to the public. Also, especially as data is disappearing, as you know, from public servers.

[59:36] So adding that to that. Yeah. Sure. Data. Right. You've had access to data science or history. Other people may have history seminars. And if you were going to give one assignment to those students, could they, with the power of LLM, contribute something that would be useful for such a problem? Could they recreate some part of the analysis themselves? Yeah, absolutely. I mean, I don't want to recommend anybody do anything illegal, but if you were to happen to have a full PDF of an edited edition, for example, of the Icelandic sagas, which are available in various repositories, you could take those and feed it a page even of just 100 records of the database that I created. and say, add all of these from the Icelandic sagas

[1:00:41] into... And why the Icelandic sagas? I've actually talked to the person who edited the Icelandic sagas, who has absolutely no interest in putting them into a database because she thinks that these things are flawed. But why? Because of the Icelandic low. Because it's the place where weather happens. It's where the Icelandic low pressure system is. And so it's extremely sensitive. There's also enormous volcanic forcing happening from Iceland. So when you actually record those things, you know that we don't have a really reliable or centralized record of all Icelandic volcanic eruptions? We don't. It doesn't exist. There's no historical one, that's for sure. The natural ones are bad. And there is no tephra. The tephra, that is the volcanic glass that comes out when a volcanic eruption happens. It's basically a fingerprint of an eruption. No catalog of that. These things make the Little Ice Age happen, and

[1:01:46] we have zero, nothing about it. So, you know, starting there would be great. I've actually talked to people at the University of Iceland about it. Not particularly receptive. Would be actually really cool to host it in Iceland Speaking of data centers, they have it. So, yeah, I've thought about it. Yeah. Thank you so much. Thank you. Thank you for coming. Thank you online as well.