Abstract

Edward J. K. Gitre and Samuel Backer each present a case of error in computational history and then take audience questions. Gitre describes The American Soldier in World War II, a project that reconciled about 250,000 transcription contributions from roughly 7,600 Zooniverse volunteers using a Jaccard dissimilarity method and hand review, and that found natural language processing (NLP) tools misreading race-related vocabulary. Backer argues that error has to be understood before a method is chosen, using optical character recognition (OCR) and Gensim topic modeling on the Lester S. Levy sheet music collection. He also reports work with Louis Hyman showing that historians need precision more than recall, and a failed attempt to extract ledger data from theater booking records with a large language model (LLM).

Chapters

Transcript

AI-generated transcript. It was produced from the recording by speech recognition and cleaned up with AI. It has not been edited by the speaker and may contain errors. The captions on YouTube are AI-generated in the same way. Each timestamp links to that moment in the video.

[0:15] here with us today, either virtually or in your meat suit. My name is Sam Backer. I'm an assistant professor of history at the University of Maine. And my name is Ed Gitre, and I'm an associate professor of history at Virginia Tech. As you can see here today, we're going to be talking about questions of error and failure and hallucination. And these are, I think, particularly pertinent, as you'll hear in some ways in relationship to AI, but also in relationship to kind of the broader body of computational approaches to history. And we're kind of, I think, going to be moving back and forth between more generative forms of error and hallucination and kind of larger computational forms of error and hallucination. And we think that these are kind of core, these are core issues and core questions for a lot of this new emerging field that we've all been

[1:15] hearing about and talking about, right? That these forms of error and how we think about the results that we produce, how error plays into those results, how we can figure out how we can trust those results, all these are absolutely vital questions. And so I think this is kind of structured as a conversation. We're going to be sharing some of our own experiences along kind of parallel tracks end with kind of a more open-ended discussion. Yeah, that sounds great. The other thing is I don't have a clock anywhere. Okay. And we definitely need a clock. Okay. Yeah, I can start a clock right now. Okay. This is a finely honed machine of a presentation. No, it actually is. We've practiced. Okay, so I think. So I'm up first, right? Yeah. So part one, error. Okay, so, and then you have the notes, and I don't think that I'm going to need them at all, but just in case. Okay.

[2:17] So, again, I'm at Virginia Tech. Among other fields I'm involved in, the one that I've done the most work in is digital history of World War II. I'm going to tell you really briefly about the project, the American soldier in World War II, And I don't have the website on here, but it's just all lowercase, American Soldier, WW2 with a digit dot org. So briefly, during World War II, I think it was the only person to raise their hand when there was a question about World War II. Probably raised my background is in cultural intellectual history. I started to do digital history starting after 2014 when I started working at Virginia Tech. During World War II, the U.S. Army created a research branch and surveyed soldiers and their morale. And over the course of the war, the Army surveyed half a million soldiers,

[3:17] and it covered all aspects of soldiers' experiences of the war. At the end of the war, they preserved about a million punch cards from their surveys. Those punch cards were read to tape later in the late 60s and then turned into ASCII files that you could download from the National Archives and also access through the Roper Center. In addition to preserving those, they microfilmed about 65,000 pages of what were called free responses, which is the last page on these survey questionnaires. And those were getting microfilmed in 1947, but there was only one set of the microfilm. And I came across the collection. It was just 44 reels back in 2009. And I knew immediately when I encountered them that I had found a gold mine.

[4:18] They were firsthand accounts. They were uncensored. They were very raw, very real. And one of the things that really attracted me in the beginning were the surveys of African-Americans. And I'm going to talk a little bit about that later on. So I get to Virginia Tech. Let me go ahead and the next slide. I'm sorry. I appreciate that. So I get to a tech school and I was like, now is my opportunity to do something because after I saw it in 2009, I really had the idea for a project. So I get to Virginia Tech. This is the place. And the first thing you can see here is this. This is what these ASCII files look like when I first started working with them. I work really closely with a sociologist who knows how to import these and parse them into SPSS. learn how you do that. We also had some code books, but the idea for the project was to resurrect and make all of these sources available to the public for the first time. The Roper Center had these files, but they're really

[5:23] inaccessible. They mentioned there was only one copy of the 44 microfilm reels. So the project is both quantitative and qualitative. So what happened was I got the team that I was working with a foundation-level grant, startup grant, rather, and then an implementation grant from the NEH back-to-back in 2017 and then 19. And the website was launched in 2021. It was an extraordinary undertaking throughout. It overlapped with COVID, and that was a lot of fun. This is what the, there we go. So this is what one of those free response pages look like right there. I said about 65,000 pages, and this is a transcription. And when I had, what we did was in the beginning, oh, I'm moving off screen, is I

[6:24] had a couple of reels digitized by the National Archives, And I actually had my students start the transcription project as a class project. And we prototyped. If you want to try it, is that? Yeah. So we did the prototyping. We used Omeka. And I worked with a computer science professor. His students created a plug-in that's like Scripto. but has some additional community features on there. We very quickly realized with the grant that Omeka wasn't going to meet all of our needs, so we switched over to Zooniverse. Does anybody know Zooniverse here? Zooniverse is a really large citizen science platform. It has 3 million members. It started very much as a science-based platform, but it's really branched out, And there's some humanities and history projects. So when I started, there were already some projects there, but it's expanded.

[7:27] Here you can see that we have two different pages. These are digitized. And the goal, as I said, was to transcribe. Now, we could have just transcribed the text, but what I wanted to do was to preserve as much as possible. But what we had to do is we had to map out because for each survey, the survey pages would be different. And I worked with some computer scientists on this. A lot of this was just manual going through the surveys and mapping them out because we have both multiple choice and short answer as well as the free responses there. Okay. So here you see the output from this transcription drive. What we did initially was we had each document transcribed three times because these were really difficult documents to transcribe.

[8:27] And transcribing microfilm is incredibly difficult. At that time, a lot of the digital history transcription projects relied on manual review of citizen archivist contributions. There was some automation, but not that much at that time. But I worked with a computer science professor, and we used an algorithm, a Jaccard dissimilarity method, to try to find the best transcriptions from the responses from our citizen archivists. there were 65,000 pages. In the end, we had 250,000 contributions from about 7,600 transcribers from around the world. So this is where it gets interesting. So I was so excited. I was like, oh, we can use an algorithm in order to pick the best transcription that was submitted. Great idea, but the implementation was more difficult. If you know that method at all, you have to have really robust data.

[9:30] So as we were manually reviewing those, we realized that we needed a whole other round. And even then, after we did that round with 250,000 submissions, it still wasn't good enough. So I engaged with about 30 transcribers, top transcribers, and they manually reviewed. So the algorithm did well, but not good enough. Okay. And I'm flying through here. I think that's me now. Okay. Yeah, so just really briefly, it doesn't show the next slide on here. So I worked with a couple of different teams. I come from an interdisciplinary background. How many of you are coders, but you're already coding? Okay, so I'm speaking to everybody else here. So my training was in my professional development was very much influenced by

[10:32] interdisciplinary settings. So I did a doctoral fellowship at the University of Virginia and that sorry at Rutgers. And that one was dominated by English professors. And then I went to UVA and there I was in a postdoc program and mainly sociology. and when I embraced digital history, I was like, I'm not going to give up that practice because at that moment, the big push was for everyone to code. So if you're a historian, learn how to code. Things have changed with AI. But back then, it was like you actually had to learn code. So I embraced interdisciplinarity, and I'm so glad that I did. In the end, the team on the project, I'm watching about 100 individuals, students, contributors from the community, but also lots of data and computer scientists. And I'll show you some of the results for the work that they did. But needless to say, the advice I had been given in the beginning that

[11:36] DH projects are 95% data cleaning. That is so true. So, so true. that I learned early on was not only the value of prototyping, but to allow to be patient and to give yourself the time with the data cleaning and restructuring and structuring over and over again. Now, I won't give you all of the gory details, but the only reason the website looks as good as it does is because of the attention to the data cleaning. Lots of manual labor, but there was AI that was used. Okay. So we can talk later about whether historians might still need to learn how to code. I'm trained as a historian in a PhD program here at Johns Hopkins, but taught myself to code during the pandemic and do actually think that it can be useful. So kind of building, again, this is a lot of kind of parallel paths on similar types of issues that I really do think are fundamental to a wide variety

[12:40] of these computational humanities structures, these digital history projects. So a question I've also grappled with is, what do you do when you're working with materials that you know are super-duper messy, which is maybe everyone's materials? And I think there's a lot, there's different strategies, philosophies of how to deal with this, right? Really fine-tuned cleaning is one of them. And I'm going to advocate here something quite different, which is in trying to make moving error and messiness from a problem to a quality of your material, right? Moving it from a, to paraphrase Donald Rumsfeld, an unknown unknown into a known unknown. And then strategizing how you can deal with the errors and issues that you know are going to be in your materials.

[13:42] And here I think there's also very real differences between different types of historical archival stuff. Right. Ed is talking about the experience of soldiers during World War Two. Messing with it. Right. Is messing with someone's grandparent or great grandparent's experience. There's a lot of stakes there. I work on popular culture, and so my data is messy. It's a vaudevillian was actually in a slightly different place than I think they are, or this lyric in this song is actually different than what I think it is, which can still cause real problems, causal problems for my historical analysis, but doesn't necessarily have the same moral charge. And so thinking through, right, there's not a universal applicable strategy for all of this. It's really different types of history, different spaces of history, who's

[14:43] going to have different kinds of strategies. So the way I've been thinking about this, I'm going to talk through this via a project about sheet music, about 19th and early 20th century sheet music that I do as part of my book project. And so the way I'm thinking about error in this, I've got a kind of a three-step schematic here, right? Step one, know your data. Step two, know your method. And step three, contextualize your results. So step one, know your data. I did this project on the Lester S. Levy sheet music collection it's collected by a very wealthy and somewhat idiosyncratic sheet music collector who gave his collection to Johns Hopkins and his family which we thank for endowing a curator for the collection round of applause for the Levy family, thank you if you're

[15:49] watching this, give as much money We love your collection So it's this amazing collection of high resolution Digitized pieces of sheet music Sheet music, though, is a nightmare for OCRing As you can see here, this is the piece The Traveling Man A random piece of sheet music that I picked Totally at random to give you guys a sense of what can go on So this isn't even with a bad printed copy or hard to read the text, which is what I'm particularly interested in, has all kinds of issues, right? It's split up by syllable, which you can sometimes address by cleaning and sometimes not. And the lines across, right, break up the semantic information. Because almost any model, unless you do a ton of work, is going to read this as quick who register their names and often register a train who lugs around two heavy grips and all the snow and kick, right?

[16:50] Versus how you're supposed to read it, which is who registers the names and often registers the kick when you start up in business and you don't know what to buy, right? You follow the music with multiple rounds of lyrics printed. The point being that I knew that this material was going to be a chaotic mess for any kind of distant reading, kind of data analytics I was going to be doing. Which leads to two. Know your method. I learned this the hard way in a pre-Chat GPT coding environment, spending a lot of time doing something simple that worked, and then an even longer time trying to do semantic embeddings that didn't work. So the method I used was basically, we used a very simple form of OCR to analyze the documents. We've used fancier forms of LLM-based multimodal language OCR. with the problem that well, and then try

[17:50] to analyze it using Gensim topic modeling, which is a very old-fashioned, for those in the topic modeling biz, it's a very old-fashioned form of topic modeling. It's like so 2010s. But the thing is that modern forms of topic modeling, most modern forms of kind of getting distant reading, involve semantic information, right? Using models that embeddings that base the meaning of words on the words that are around them. But as we just said, the format of sheet music shreds all that semantic information and in fact often puts words seemingly next to words that they're actually not next to. Which means that fancy forms of data analytics produce really terrible results on this material, right? There's too much error. But if you know the kind of error you have, you can use, Gensim uses what's called a bag of words approach, which just looks at all the words that exist

[18:54] together in a document, right? And that actually, because I knew that there was semantic error, you could pick a method that addresses that, right? Introduces other kinds of error, maybe, in that it is not as exact on a really high -quality corpus as more modern machine learning methods, but which is perfectly suited to what the actual archival material I have is. And then finally, contextualize your results, right? And this, I think, is a crucial problem that we're still really thinking about and dealing with in computational history writ large, which is we have these results. And we know there's error in them because we know that it's based on messy humanities data. We know that there's issues with the analytic methods we're using. How do we bring them into conversation with the kinds of historical narratives that we're all trying to write? if you just keep it in kind of

[19:57] a science-y space, a CS space, a data analytics space, what you can do is specify all these things. For this corpus, with these methods, with all the settings turned this way or that way, I had this hypothesis and it produced this results with this potential avenue for future study. And that's great because of all of those kind of hedge around the conclusion that you're making. It doesn't directly map out onto real historical phenomenon that occurred in the real world and bounce up against all the other real kinds of things that people are doing now and in the past. The problem is, though, when we pull this into historical work, and when we take, let's say, this is a selection from this topic modeling that I did, and we bring it into this broader narrative about cultural change in this period. And this is where I think actually historians are really well, already very well

[21:00] suited for doing this. Because we do it all the time. You find one first-person account in one archive. You don't say, oh, that proves it. It's exactly like this one first-person account says or this one soldier's letter says. We know how to triangulate between different archival materials, different sources, different kinds of sources, and kind of read them against the grain. And that, I think, is the final step here, right? Is to think about how we can bring these quantitatively based sources into conversation with all the other kinds of things that could tell us about, in this case, what's going on in the relative distribution of topics in sheet music in this period of time. which is composers' biographies and newspaper clippings and business records. And when they line up, you go, oh, this evidence supports these other forms of evidence. And when you don't, you have to say, okay, one of these isn't lining up

[22:04] right. And that kind of error is actually something that we're really good at dealing with. And I think is one of my final takeaways, all of these kinds of error, historians are really really good at knowing that most of our materials are missing most of our sources are lying and so actually thinking about the kinds of error created by these quantitative methods is just another example of something we already spend like 60 percent of our time dealing with uh rather than some radically different thing allows us to start i think getting maybe a on this problem. So part two of hallucination. Okay. So thank you for reminding me about audiences. So when I embarked on the project, there were kind of three tiers. The first was to create this digital archive that was faithful to the records. And so while we used an algorithm to help us select the best transcription,

[23:09] for me, the standard for quality was really quite high mentioned, Sam, there's a lot invested in this history, and I wanted to honor that history and the individual's lives. But on top of that, there was the question of how to make sense of it, how to present it to the public, and what tools are available to do that. And then on top of that, as a third, and this is my own particular interest, is how does this change historical methods and research. And I'll show you an example of that in a moment. But I wouldn't call it necessarily hallucinations, but under the broader topic here, what I did was I collaborated with the Computational Modeling and Data Analytics Program at Virginia Tech to see what the data scientists could do with all these transcriptions and annotations from the public who transcribed improve the website that we were creating.

[24:10] And I'm just going to highlight a couple of things that actually made it into the final website, which was to work with tags and then also calculate an interest score. So I worked with a mathematician in this program who, His name is Mark Embree, an amazing, amazing mathematician. And what he did was we worked with teams. There's a capstone program within this program. And so this was the work of undergraduates at Virginia Tech. And I'm not going to get into the nuts and bolts of this, but basically what you do with the vector language embedding is to convert words to numbers and graphs. this was quite new at the time. The really popular model for doing this was Word2Vec, which was to convert

[25:12] these words into vectors. And then what you do when you have them into vectors is you measure the angle. And that helps you to find relationships among words, between words. The first one was in 2013. And then BERT, both of these were developed by Google in 2018. I got that implementation grant. And the reason why we worked on this is that I provided 166 expert tags to those annotators, but they were also able to contribute their own tags, and they did a lot. There were over 4,000 additional tags. And so what we did was we used this method in order to consolidate those tags. And this worked fairly well in cleaning up the tags, but it got more complicated when we tried to find representative surveys. So we added to this vector language embedding, what you see here,

[26:18] interpolatory decompositions. I am not going to get into this, but what we tried to do was to combine two different methods in order to find surveys that were particularly compelling, that we could show on the website the strength of the collection. This did not work well. And this is another one of the lessons that I learned is that there was a difference between workflow that was critical to the project versus experimentation and learning. And there was a lot of experimentation and learning with the project. This is one example where we had a lot of difficulty finding But we worked with another student in order to refine it. So if you go to the website right now, the order in which surveys are presented is based on this particular method to find interesting surveys.

[27:19] I want to see it refined. At the time, it worked okay, but it wasn't great because it would pick out. Here's an example of representative responses. And this was representative. I haven't got any comments to make where Soldier didn't really have much to say. And there are lots of these, but it's not particularly interesting. The same thing with the tags, but it was very similar to that. So we did use it in the presentation. And it's doing something very weird. I also worked with the Biocomplexity Institute at UVA from another aspect. And this goes to that third level of using artificial intelligence in order to inform historical research. So my particular interest in the survey program now and has been for a while is on race relations. They surveyed black and white soldiers about race relations.

[28:20] And so we used, and you can see the results here on the right, network analysis. And I worked with data scientists, as I said, from UVA in order to understand race relations. And I'm just going to highlight one takeaway from this, which is how many people know about double consciousness? Du Bois, double consciousness? Okay. So as we were graphing, we were working with different configurations of our networks. And if you'll notice here, we have white and Negro and black man here, but there's no reference to white there. And I thought it was very interesting because this visualized that double consciousness. As remember, as we were working as a team, I was like, I know it exists, but actually to see how central the experience of race was for soldiers, black soldiers here was really illuminating. So we have this published in Current Research in Digital History,

[29:25] and you can find that article now. But there are still lots of problems here. So I'm just going to point out one word, and then I'll hand it up back to you. Negro. Is that a negative word? Does it have negative connotations or positive connotations? At the time when we were working through the collection and using NLP, if you took the word white and compared it to black, white would be tagged as positive, black as negative. Think about the word Negro. For us, you wouldn't want to use the word at all. I mean, it's not as bad as the other word. But for African-Americans, a lot of them, there was pride in using that term Negro, and it was commonly used. How do you use NLP in order to make sense of this corpus when it is so largely defined by LLMs?

[30:25] Where do the LLMs come from? Google Books. That means the majority of those books are self-selected. I mean, they're by white publishers. And so it's overwhelming bias. And I'm not the first person just to note this. well known. And another issue was of cursing. So African-American soldiers got increasingly more angry as the war went on. And by 1944, there were fears of a civil war. There is a lot of cursing in the collection from the African-Americans. And I remember talking to my colleagues, the data scientists, I was like, okay, so what techniques do you got to help make sense of the and it was like a blank look. They were like, I do not know how we deal with this. Now, that was about five years ago. Things have changed, but there's still an incredible amount of bias in

[31:27] AI. Okay. So I think there's a whole video. Oh, there's more. Do you want to talk about this? Yep. No, that's good. This is an example of one of the, this is one I was going to give you here of some of the languages. This isn't the most, but here's another one, cracker. Cracker, you know, food? No. So this is an African-American writing about whites, but you can see like, how do you make sense of this? But back to the error, if you go to the website, what you're gonna see is you'll see transcriptions that looks nicer than this, as well as the image for verification. So even if there are errors made in the transcribing of the documents, you yourself can verify by looking at the original images. Okay, left to you. Okay, so this comes out of work from a paper that I published with Lou Hyman, who many of you saw talk this morning. And it is another approach to thinking about error.

[32:29] And thinking about error both as in the error that's introduced or hallucinations that are introduced by digital computational methods, but balancing them against the other enormous points of error in traditional historical practice, right? Not reading huge swaths of the historical record brings in error, right? There's large-scale historical phenomena that we're just not able to really engage with because they aren't particularly well-suited for the more traditional tools of archival close reading. That's also a form of error. And so thinking about those tradeoffs helped Lou and I think through some of the tradeoffs in relationship to OCR, which is more and more solved within certain types of materials and certain setups. but also for really large-scale documentary records can also really be a

[33:33] significant financial burden, right? There are still real barriers for under-resourced languages, for folks who are reliant on black box models, for all kinds of things, right? So there's this kind of classic triangle of cost, speed, and quality, and you can only have two of the three, right? It can be – you can have low cost and high quality, but it's going to be really slow. You can have high cost and high quality, but it's going to be – sorry, you can be high speed and high quality, but it's going to be expensive. You get the sense. And so what we were trying to figure out is ways that we can address some of these issues. and one of the really exciting spaces i think of technical development is layering multiple forms of llms or computational methods on top of each other to get better results there's all kinds of neat work being done in this adversarial interactions um using this to deal with new problems of sentiment analysis where

[34:36] you get summaries and analyze the sentiment what we were working on was kind of ocr and in particular at the time we were doing this research it shifted I think there's another year where I can still give these slides before I have to completely revamp them. Is that you could use the multimodal large language models to get really high quality OCR results. But it was pretty slow and it was relatively expensive, especially over a large corpus. And so what we wanted to see was whether we could use a multimodal large language model. like the top best cutting-edge multimodal large language model at the time or now, use it to produce artificial but very good transcriptions, and then use a smaller amount of that material to fine-tune a cheaper model to run over the larger scale of the material, right? That will allow you to get kind of the kinks and specificities of a specific

[35:36] archival collection, and then maybe it would allow us to fight our way out of that triangle, which with varying levels of success, it kind of works. You do get an improvement, but probably not strong enough improvement that this technique right now is going to be picked up. And I didn't make a million dollars from this research. I'm really sad to say. I don't know if there's a million dollars in OCR, but maybe there is. But one of the things we learned that really stuck with me through this project again, thinking about the broader dynamics of hallucination and error, is what we spent doing, the time we spent kind of grappling with this clear disjunct between what the fine-tuned models we were producing did, and we were looking at the results and being like, these look a lot, this is kind of working, these are looking better. and then running kind of classic machine CS

[36:38] analytics on them. And they would be like, you've made it worse. You've made it significantly worse. We were like, we don't understand why is what we're seeing as historians, why are what we're seeing so different from what the results of the tests are telling us? What we realized was that what we were seeing, we were seeing as historians. And the tests were designed, came out of machine translation. What we realized was that the rubric, right, the rulebook for judging how well a OCR method worked for a task that's defined by machine translation and one that's defined by historians actually looks meaningfully different, right? And that there are certain kinds of error, again, that historians are okay with or relatively okay with. certain kinds of error that are an unmitigated disaster. The standard way to judge these metrics is Levenshtein distance, is how many changes

[37:39] does it take to get from the correct translation, transcription, to what the model put out. What we realized actually was that there were, that didn't accurately capture what historians were used, are interested in. Most historical is missing, right? I work in theater history a lot. Theater's all burnt down in the 19th century. Most of what I want to look at is ashes, a long time ago. Medieval historians, most of the stuff they want to look at was never written down. Or if it was, maybe some of it survives, right? Historians, we've developed all these techniques to deal with missing data. So recall actually isn't that important. How much of the data, the correct stuff that's in that archive, does the method bring back? It's important, but if it's a low number, that's kind of okay for historians. What's of absolute vital importance is precision.

[38:44] the sense that words that are not in that original record cannot get introduced into what the model is giving me because then as a historian i've got no tools for dealing with that really i mean i don't trust sources i can i can trust that the source is lying but not that i'm making up words as i'm reading um so that actually figuring out how to assess model result quality in a whole variety of projects, a lot of the tools and methods that come from other disciplines aren't particularly well suited for history. And that we decided, right, recall precision, moderately important, a good thing to have, recall like our drop deadline. Which kind of brings me back to the beginning point, right? which is thinking about knowing your data, knowing your methods, and then

[39:44] figuring out how to contextualize them so that you can figure out what kinds of error are acceptable to you, and it's going to be different for every different kind of historical computational inquiry, and what kinds of errors are unacceptable. So this is kind of final thoughts based on a recent kind of very ad hoc, much less organized experiment I did. So I've been working with booking records from York area theaters in the moment of early cinema, which are very important for me and not that interesting for many others. And I had all of these booking records and I wanted to get the financial information out of them. And I had two sets of booking records and the first set of booking records worked great. Actually, I had It gave me back material to double check. I ran it a second time. Then I compared the two model outputs. I double checked 10 myself.

[40:46] Whenever they agreed, it was right. Every time they disagreed on an important fact, I went back in, figured out which one was right. It was a mixture of human annotation and model response. It produced good results that I could use in my research in an afternoon. Hooray, win for AI. and then I was like well if it works for this this is me breaking the rules I've developed for this talk if it works for this it will also surely work for the next bunch of theatrical papers that I had from four years earlier so I was I condensed right I was like oh it's applicable here it's applicable there I didn't know my material and I didn't really clearly think about what kinds of error were acceptable and not acceptable I'm just going to move over here This is two pages of a ledger. They run this way across two images with weeks records bunched like this. And I spent almost two weeks this summer banging my head against the

[41:51] wall, trying to get this model to extract, accurately extract information across these two ledgers. It did it for the other one, I thought. So why shouldn't it be able to do it for this thing that like a second grader could probably do? And it just was slightly too difficult because if you mix up one of these columns, you transcribe financial information from one week and one theater into a different week and a different theater. And it's a catastrophe. Unlike the sheet music, I had zero. I'm trying to figure out whether these theaters were profitable. room for error of this kind. So having assessed that, I was A, unable to make this work. But so thinking through that kind of structure to assess pragmatically whether you're likely to have success or failure in a different setting. But also I wanted to do this to highlight another form of hallucination which is uh

[42:55] i don't have a great term of art for it yet but uh model assisted superhero syndrome maybe right chat gpt was like this is doable after a day no no we're like this is close we're gonna do we can do this we can do that we can do this we can do that i like threw ideas. It said the ideas were great. And there is this process, right, of with these technologies that are so incredibly effective for certain things, they also don't do a very good job of kind of taking a step back and approaching problems systemically. And they're never going to say, oh, no, actually, like, this seemingly simple task is just too hard. So that last level of kind of mission creep that comes from working with these computational tools is another level of hallucination, I would call it, that we also have to be aware of and a source of error that we also have to be aware of, right? If I had been more attentive to that, I would have saved myself.

[43:58] I don't even want to talk about or consider how many hours in the two most frustrating weeks of my life, perhaps. But, right, so thinking about all of these sources of error that don't just have to be within these very limited structures, but that can come from this whole pragmatic set of spaces in which we're using these LLMs to kind of instantiate new computational methods. Yes. Okay, so there... So I think we've got 18 minutes. Well, do we have 18 minutes? You get 11 minutes. And so let's sources of error. So I think, you know, we were supposed to have a back and forth, but there are lots of people in this room. And who's monitoring for online questions? Do we have anybody who's doing that? That would be fantastic. So I would love to talk to you, but I also want to I definitely want to have time for Q&A because I just spewed out.

[45:02] I condense so much and I'm sure that I rush. So I would like to take Q&A. Yeah, absolutely. But I agree. It's interesting to hear your story because a lot of my project was before generative AI. And so when I was working with these algorithms, it was just assumed you had to verify everything. You couldn't just assume that the output was great. You literally had to go through manually. That was just part of what you did in the data sciences where I found myself when I was pumping information in, there was a lot of optimism. But then when you actually do the verification, it was like, oh, no, this is garbage. So I was in that earlier period. And so you just had to spend a lot of time with the mapping of data, structuring of data, because it would just fail, literally would fail if it wasn't well structured rather than we can work with this. and then also the verification okay so uh let's open the floor

[46:07] so this is finesse to see um for a kind of bit on a g student top insights uh fifth year abt i've gotten more and more into these tools over the past six months this time a year ago i got a bunch of documents in russian and the language i read from many members stands in february 2022 before the new moscow very quickly and I think for all experiences with ChatGPT or Gemini, in the chat window, pre-agent era, complete garbage. It would stop after a page or two, hallucinate, terrible stop. Agent era, a bit better, but even in bark, I was experiencing large volumes of hallucination, totally corrupt, totally garbage. It's just gotten unbelievably better in the past six months. So I'm curious where you see, as you said back in the day, you couldn't even count the output as legitimate. You'd have to manually verify it every time. My rule for myself, because I'm looking at something like 2.5 million digitized files from HW. Pointing errors, anything that I'm going to quote in publication, I go back to, you know, this is page 785 in the PDF that's collated.

[47:09] I verify site to site, and I do that as well with documents I scan. But I'm not assuming that some broader summary is itself based on the progress I've seen in the past three months. Do you think that's a reasonable position, or what rise to the level of you must verify? text of the thing i'm publishing that guys look at the actual page go to the pdf otherwise for larger characterizations i'm a bit more trusting now what do you think so i mean i think that i mean it's a great question and it's like it's one that we're all going many folks are going to deal with more and more and more um not everyone but like people working history in the 90s like 80s and 90s like yeah um so i i think that there need to be systemic ways of assessing corpus level, the validity of anything you're doing corpus level. That should probably go through a couple of different types of approaches because, again, error can come in all kinds of

[48:11] different ways. I don't know the specifics of Kremlinese in order to figure out where those might be. But as a kind of top of the mind reaction, right, sampling yourself, like, and I do this for more serious parts of my research than this project was, random sampling yourself a meaningful number, like a hundred randomly selected pages, because what can easily happen is it's really good quality images, except for this one chunk that you don't actually ever get to. And then that chunk is meaningfully lower quality photos. and actually the OCR is much worse there, right? Then probably some sort of like corpus level tests, right? Keywords, stuff that's just like kind of a reality check. Like is this kind of giving you the results that you think it should, given your understanding of the historiography of the era? Just like making sure that this

[49:12] archive is okay. And yeah, I think that we're gonna get better at figuring out the tools for doing that. But I do think that that kind of corpus-level work and also yourself just spending some time really digging in and verifying across it is probably necessary before you just trust it. Because, again, those unknown unknowns can come from all over the place, and we precisely don't know when they hit. Yeah. My perspective on this is that in the science, they quantify error and define what is an acceptable rate of error. incredibly high standards for the evidence and how we use evidence. I don't see where this goes unless we were, say, to embrace some of those standards that are used in the sciences to quantify how much error is acceptable so that people who are rewriting the data through your models or trying to replicate, can expect the same

[50:17] rate of error and find it acceptable. Like, we don't have that at all. And so I don't see how even, I mean, these are going to, maybe someday it's going to be foolproof, but I think we are so far off from that. I think the only middle ground would be something like that, where you're actually quantifying the rate of error so that people who are rerunner know that this is your rate of error. Because you can, I mean, with transcriptions, you can do that. We didn't do that because the standard was so high. for what we wanted. But that's, I mean, that's the only thing practically I could see in the interim until we get to that point. And just to add on to that, I also think that's a space where computational humanities practice can feed meaningfully back into traditional archival practice. In my ideal utopian world, everyone, every archive you went to, you would write at the end of the book or the article, a little description of what the archive was, how you used it, how much of it you got to look at, were there parts of it you wanted to look at that you didn't get to look at, did you do any sampling, which

[51:18] could, you know, coming from this wonderful book by Claire Lemercier and Zalc, just like picking some random boxes and picking randomly in those boxes just to see like even a rule of thumb sampling of it, right? That I think that the same kind of error can exist in traditional archival methods and and this kind of rigorousness would be good for history at large. Another question. Thank you for this. I have a question that sort of jumps away from, but still it's on what I'll call a methodology question. What are the standards for research? My question is about all the other forms of technology that are implicated obstacles in your research, you let off by talking about tabs and reels and microfilms. And so I think of the material challenge side of archival work of people keeping things, losing things, things being produced to

[52:24] ashes. And I'm curious about how the standards of research conversation is intersecting with the obsolescence reality of her struggle of keeping things and being able to recover them. And I am a history graduate student here at Johns Hopkins. I do not have a computer science background, so I do not understand the technologies of LLMs enough to articulate the question the way I want to, but since you have experience dealing with different systems of storage at different Yeah. What what are we dealing with 20 years from now when the LLM technology changes? What does it do to the research that's being produced now? And how on earth am I ever going to access something on a floppy disk? Yeah. There's also been a request to have you guys looking questions. Oh, right. Thank you. We've totally forgot that. So sorry, Chad. Yes.

[53:28] The intersection of LLMs with obsolescence or just the future? That's the question. I'm curious about your present experience of needing to make very different technologies speak to each other. your research now. And I'm also curious about your speculation, given how rapidly this technology is changing, what that means for my research 20 years from now. In many ways, I think that the digitization has not changed that much. I mean, National Archives and others, they are literally digitizing, and a lot of their efforts are on digitizing microfilm reels. The technology is there for doing that. Internet archives is doing a ton of that as well. So I don't see that changing anytime in the near future. And there's technology for doing that work of conversion. I think the next step of how you extract the information from those digitized

[54:30] sources, because that's what we ran into, I think that will change more rapidly than just the technology to convert from older media to digitized sources. I think what's interesting with LLMs, these companies know that they're limited by the corpus from which they're working. And so for them, it is a boom to have all these archival sources digitized because basically it makes for more robust LLMs. And so I see an investment on that side of converting these sources some of the limitations of just working with books. So we have another question. Yeah. And I think this last question. I mean, we're happy to stay if people wanna. Yeah, definitely feel free to stay and ask us more questions. I mean, we're counting the two sides.

[55:31] And I will say that when I actually talk to ChatGPT or ask ChatGPT to help me or any of the AI labs, there are all sorts of barriers around talking about suicide and they immediately tell me oh my lady i cannot address this i can get around it with some air like saying i'm writing on history some ai models will not take that as the answer i'm kind of curious what you think about this kind of bar rails that are being put around ai and how ai thinks about topics that are considered we're talking about sensitive, thought of how much I should almost avoid AI because I'm going to guess that it's leading out huge chunks of things that I would love or assume that it's open. Yeah. I mean, I would just say that based on my experience right now,

[56:31] in terms of the interpretation of the sources, there are things that i can do with lms with the volume but when it comes to what you're talking about i don't i don't see a ready solution to that a solution at hand the the the the thing i i do think though that is um for me i think and one of the a huge get with these tools because i think there are, back to the previous question, about a lot of the cutting edge models are black boxed, Anthropic changes things, your result changes, right, if you rerun the experiment. And so thinking about using these as coding agents that can provide reproducible, less mechanically deterministic forms of data analysis that you can iterate on pretty quickly, I think in many ways is really, for me, the obvious get for using LLMs in history, right?

[57:33] So less read all these documents, tell me what you think, and more let me apply embedding whatever on a huge corpus. Let me use that to figure out the kinds of documents that deal with the topics I'm interested in and then allow that to surface the documents back without necessarily using the generative aspects. It doesn't sometimes like to do anything. topic yeah because i used the nlp it was not yeah right yeah it's just it's like a barrier which i understand um but i wonder because i'm guessing that other topics that people have to do such similar yeah ai is just not going over yeah so i think we're feeding documents yet coding might be your way to do that then well thank you collaborate collaborate with people yeah Always connect. Yeah. So we're just sorry for three minutes over. Happy to talk to anybody. You can also email.

[58:36] Like I said, I just just ran through a lot. We both said anyone on chat. Thank you so much. Online folks, email us or if in the discord, we will answer them. Thank you all so much. Thank you for coming.