Abstract

Humphries walks through how large language models (LLMs) handle historical language and handwriting at scale, drawing on his record-linkage work with 10 million pages of digitized First World War veterans' pension files and an ongoing fur-trade project with Lianne Leddy. He argues that models such as Gemini now show something like abstract reasoning, citing its transcription of an unpublished Albany ledger, and reports a word error rate of about 1.8 percent on 18th and 19th century English handwriting. He demonstrates the Archive Pearl site for transcription, segmentation, metadata extraction and map pin placement, and warns that models are least reliable at reporting negatives. He urges historians to benchmark models on tasks with known answers and to build verification into every stage.

Chapters

Transcript

AI-generated transcript. It was produced from the recording by speech recognition and cleaned up with AI. It has not been edited by the speaker and may contain errors. The captions on YouTube are AI-generated in the same way. Each timestamp links to that moment in the video.

[0:06] Hi, everyone. I think we're just about a minute away from getting started here as people continue to file in virtually. We'll start it here in just a second. All right. So hopefully that will look like I'm going to put the chat up here so I can see what's going on. If there's any problems with the layout or anything as we go through, just put it into the chat because that's what I'm going to have up here. But you should hopefully see me on one side of your screen and the presentation on the other. I'm going to flip back and forth between the two as we kind of go through this. My name is Mark Humphries.

[1:07] I'm a professor at Wilfrid Laurier University in Waterloo, Ontario, Canada. and I'm sorry I can't be there in person but I'm very happy to be presenting here virtually today. What I want to do is talk a bit about you know some tools that I've built, the approaches that I've taken to using text in history but I think it's probably a good idea to at least start with kind of a discussion about where I've come from and all of this. So for myself I'm an historian topics over the years. I've written about the 1918 influenza pandemic, about shell shock in the First World War in Canada, and then during the pandemic I kind of became interested in family history and I started working on the fur trade and that's taken me down a completely different road in the 19th century. But for me all the things that I've worked on have been basically focused around writing histories of ordinary people, micro histories.

[2:11] And what I've found in doing this over the years is that it is the best stories we can find to write about people and the information we often need to find out about people is buried in mountains of data. I'm old enough to kind of remember the days of going into archives with just a pencil and paper before in some of our Canadian archives, you could even bring a laptop in. So I've lived through that whole digital light revolution where we take pictures and do things like that with archival documents. But of course, this just created mountains and mountains of data for us to sift through. Myself, I really began to come to AI as a way of handling some of this via another project that I did when I was head of a research center here. We digitized 10 million pages of veterans pension files in Canada, which ended up just being JPEG scanned off about 1,700 microfilm reels. And the problem was none of this material was standardized.

[3:12] The forms changed constantly or weren't used. And so going through them and just indexing the material took about eight years and about 30 different students over the course of that time. Looking back now, had we had AI to do some of these things, there'd be a lot of work that we could have done that we wished we could have done that was impossible at the time. The project I'm currently working on with Dr. Lianne Leddy here at Wilfrid Laurier University, who's a Canada Research Chair, looks at a similarly voluminous kind of corpus of records. We're trying to trace fur trade families through a huge group of archives and look for individual people in places where you don't know they're going to find them. AI then for us has been kind of the way we've been able to kind of envision doing these projects. And what I want to talk about here today is how we can use AI to do these types of things.

[4:16] Just to give you an idea of where we're going to go in the webinar here today, I'm going to start talking a little bit about LLMs in general. I'm going to assume that most people being here know how they work and we're going to skip some of the technical stuff. I'm going to of things that are important for thinking about how they work with text and what we need to do to get good historical results from the models themselves. I'm going to talk about the way in which LLMs have changed how we handle things like image processing, transcription, all of these types of things, and the stages you have to go through, at least in my experience, in order to ensure that the results you're getting at the end are usable. And I'm going to do a demo of what this actually We created that's an open access or a not-for-profit website run through Wilfrid Laurier here that people can go on to use some of these tools that I'm going to demo and describe here today. I'm also going to talk about the limitations that we have to work with here. I'm going to assume that, you know, discussions about the

[5:18] environmental impacts, ethical considerations, copyright, all those other things. I, too, understand that these are issues. I'm going to assume that other people in the conference are going to focus on those. I'm going to talk about here are methodological limitations and things we have to keep in mind as we do research. I'm going to propose a few important rules, I think, that I've come up with, at least, that I'm currently drafting an article on to think about what we need to do with AI results and how we need to use them. Because of an article I published in Historical Methods a couple of years ago, I'm kind of getting a lot of requests to review papers on AI methods in history these days. And one of the things that I find is that there's a lot of assumptions that people aren't making about how to use this material. And it's leaving me a little bit anxious about what this is going to look like in the future. And I'm going to talk very quickly towards the end about where things go from here. Hopefully, we'll have lots of time for questions as well. I know that the session is slated for an hour, so I'm going to try and keep that in mind as I go through.

[6:22] Maybe I'll just do this and make this a little bit easier. Hopefully, that works So, obviously, large language models are created, are based on neural networks, right? And neural networks themselves are modeled on loosely, at least on the structures of the human brain. And we know that they are trained on an enormous amount of data. I'm also going to say that I think people understand as well, hopefully here, that large language models are not databases. They don't contain all that text that they have ingested during training. What they contain instead, to put it succinctly, is the probability that that text exists given a specific type of an input. These models are trained on next token prediction, right? Which means in training, they are trained to predict the end of a sentence like the cat set on the blank. And they do this trillions and trillions and trillions of times across mountains and mountains of data, which creates the structures of the

[7:24] network itself. The math behind that is crazy. And I do a lot of these types of discussions where I spend lots of time talking about how this stuff works. I find it fascinating and interesting. But suffice it to say here that the thing we need to understand in using these models is that the premise of generative AI as the industry exists today and the technology exists today is that in encoding language this way, you can actually encode the structures of human reasoning and that networks that are trained on that language are able to replicate those structures of reasoning and then perform the types of general computational operations that we associate with intelligence. That's a very complicated way of saying that the assumption here is that next token prediction is a means to an end. It's not an end in itself. And that training the models in this way produces a model that is able to emulate thinking. And that the idea is that those patterns of thinking are embedded in the structures of language. This overturns a lot of thinking about how language works.

[8:29] Whether or not this ultimately proves to be true or not is still somewhat of an open question, but the reality is as these models continue to get better and better at doing the things that they do, it's becoming increasingly evident that the models are certainly capable of emulating intelligence and reasoning. the rationale for why that happens is kind of something that may be up for a little bit more debate still now do they actually reason well I first kind of came you know I mean I was I've spent a lot of time thinking about this I'm going to flip to this view here now I've spent a lot of time thinking about this over the years I first kind of came to the conclusion that in fact yes The models do reason last fall when I was working with a yet unreleased Gemini model. It turned out to be Gemini 3 and later 3.1. And to this point in my career with AI, we've been developing tools to basically

[9:32] use AI on handwriting transcription. To this point, models were not very good at this. You could use Transkribus as an example to transcribe documents. You had to do lots of fine-tuning with Transkribus to get it to do what you wanted it to do. And then you could initially get large language models to do some transcription of the text itself from the vision with vision. And they weren't all that accurate. You could then take them and have large language models correct the text that Transkribus produced in order to produce a more accurate text. And then along came Gemini. And what surprised me with Gemini when we were using it was that we found that it was highly accurate even on images like this, which are difficult to read and fairly messy. And while that was interesting to me in terms of transcription and I knew it was going to be useful, it was this result that really kind of left me convinced that we're dealing with something completely different here. So in transcribing this

[10:35] messy ledger, and it was just a page that I had given Gemini to transcribe, I found this really significant because I had captured this image myself in a small archive in Albany. It had not been on the internet before, so it couldn't have appeared in training data. I was able to confirm later that it hadn't. But, you know, it was new to Gemini. And as it went through, it took this text, I'm highlighting it here, and it translated it or transcribed it as entered Samuel Slit 27th, in this case of April, in Albany to one loaf of sugar, 14 pounds, five ounces at one shilling, four pence, zero pounds, 19 shillings, one pence. What's significant about that is that if you look at the original above, nowhere does 14 pounds, five ounces appear. You get one loaf sugar, there's a bit of a, it looks like a quotation mark or perhaps a small hash or U, then $145. And at first I looked at this and I thought it was an hallucination. And then what I realized was that the model had actually done some really interesting abstract things here. It had taken $145 and concluded

[11:41] that $145 was in fact 14 pounds, five ounces. Now, to do that, you have to actually figure out how to divide pounds, shillings, and pence by the unit price and then get the decimalized number you get as a result of that equation and convert it into a two-base system of pounds and ounces. And there's a lot of stuff going on here. You first of all have to realize that the 1 slash 4 refers to one shilling four pence, not one pound four shillings. You have to know how many pennies there are in a shilling and how many shillings there are in a pound. You also have to know that sugar was sold in loaves in the 18th century and that the loaf would also be accompanied by a measurement in pounds. But you would also have to understand that buying one loaf of sugar that weighed 145 pounds was highly improbable, and that in fact, it must have been something else. I looked at this and what I saw was some really good examples of what in a human being I would only be able to describe as abstract reasoning.

[12:43] And for me, at that moment, I began to realize that the models could probably do a lot more than I had thought able to do up until that point. And it really was with Gemini 3 that the model capabilities began to take off. Over the last year, we've seen a huge kind of advance in terms of what they can do. All the math problems that the models have been solving, the 700 or so problems in OpenAI dropped a week and a half ago onto the mathematics community, are just kind of one example of this, right? But the models are now reasoning. And so what can we do with this as historians? there's a lot we can do but we have to be cautious in how we do that and that's what i'm going to talk about to set it up though i think we need to understand what this current kind of state of ai looks like today uh it is um a uh it is a rapidly evolving situation to kind of uh um say the least We're moving away from a world in which AI, as

[13:46] we know it in generative AI, is defined by the chatbot. I think in a couple of years, people will look back and find it very quaint that people were chatting with models kind of in a chatbot web interface. Obviously, almost all the models now are completely multimodal, meaning they can take audio as input, videos, input, images, text, and output all of those same things. Whether or not those features are enabled or not is a different kind of a question. But what we're seeing is we move away from chatbots using AI models that are in fact multimodal is a move towards agentic AI, as I'm sure most people here are well aware. For those of you who aren't all that familiar with the term, agentic AI describes AI models that are able to take actions effectively in the world. They have tools that are written in code for them, which are part of the harness that they use to interact with us. And those harnesses are things like Claude Code or Claude Cowork or ChatGPT Work,

[14:49] the ChatGPT app effectively on your computer. They use these things in order to be able to do things in the real world. They can, you know, book flights if you give them your credit card. They can do all those sorts of things. They can write code, but they can also manipulate files. They can move things around. In other words, they can really start to become assistants within these kind of applications. All of that is happening simultaneously as we're watching the model scale ever larger. And scaling continues to hold as the so-called scaling laws predicted back maybe about six or seven years ago that they would. There's a lot been written about scaling. I'm not going to get too much into that. that people are interested. But in general terms, the scaling laws suggest that you can predict the capabilities of a model based effectively on its size, the number of artificial neurons or parameters that it has. And those continue to hold as we see new models like Astra or Gemini 4, which came out the other day, the Anthropic, Fable, and Mythos class models.

[15:52] The bigger the models get, the better they get at doing things. They're now able to do things in the real world. And all of these things are going together to mean that these models are increasingly able to master a number of tasks and whole kind of workflows in knowledge work. And that really began with coding a couple of years ago. We're seeing it play out with mathematics right now. And the reality is in the next few months to limited number of years, we're going to see this play out in knowledge work more broadly and more generally. So the first thing I would say to you as I say this is a caveat. available for people later. I know there's a lot of text there, but you know, you can take a look at it, hopefully, on the screen and later if you need to. But all I really want to say is that a lot of what I'm going to show you here and a lot of what you hear about AI is dependent on a few different things. One of the most confusing aspects of this is that there are literally dozens of different models, and there are many more dozens of different ways you can use

[16:56] those models and many more dozens of settings that you can apply within that kind of matrices, which means that no two people are, it's true that two different people are often using models in two very different ways, right? And that I keep running into situations where people will say to me, the models don't do what you're describing for me. And almost always, it's going to be a question of, well, what is the harness that you're using? What's the program? How are you using it? What information are you giving it? What capabilities are enabled? What connectors are available? Can it access Chrome directly or not? All these different things. What's your thinking level? These are all questions that make it really difficult to compare apples to apples. What I'm going to try and do here is tell you about a specific harness that I'm using and then talk about some of the more general ones that people can use. But keep in mind, as you're trying to evaluate the capabilities of models, it's not just simply a question of can ChatGPT do X, it's a question of what can the model that underlies ChatGPT, which is an application, what can that base model

[18:00] or trained version of that base model do when given the right tools in the right environment with the right parameters? And setting that up is not an easy kind of thing to do and it's not always intuitive. But just keep in mind that your own experience may vary can't do the thing that you want them to do. It may be because you found something different, but it may also be because you haven't set it up in quite the way that would be necessary to get the model to do what you want it to do. It's a really important kind of caveat. One of the most important things to think about too is the question of whether we can actually trust LLMs or not, right? And in the past, we focused an awful lot on hallucinations. And what I would say is that over the past years, 18 months, but really the last year, hallucinations have become less and less of a problem if you're using the models with a high thinking setting and you're using kind of the best available frontier models. That's that caveat from that slide a minute ago.

[19:04] The reality today is that models will hallucinate. They don't tend to be hallucinations that are often easily detectable because they will look right. And this isn't kind of like saying a citation, you know, like you used to see with ChatGPT 3.5, the original version that came out, where it'd make up a citation. It's more today like an example of assuming something that is reasonable to assume, given a lack of evidence that the model then fills in. That can still be enormously catastrophic, depending on what you're using that material for. so what we need to do is we need to find ways to minimize hallucinations and to validate and to verify what we're actually doing with them all is what we're getting them to do what i would say is that on most tasks though you can begin to trust llms if you're able to set up a clear methodology that is replicable in which you build in verification steps intentionally into that to

[20:08] make sure that you know what the model is doing. And if you establish benchmarks that you can use along the way to figure out how accurate a model is on a given task, because the capabilities of these models are jagged. Some things they're better at, some things they're worse at. What I would argue for then is the need to develop a very consciously methodological approach to using these models. This is not a situation where you just dump stuff into a model and deal with the outputs. That's going to lead to real problems. It's more like treating this as a tool that you need to understand as a tool and to treat it like a tool. And what we've done in our team is we've been building these benchmarks for the last couple of years. We started off with transcription where we created our own kind of like group of records that we knew weren't in the model training data. And then we've been able to then be able to say systematically over time, how good is a new model on transcription? We're able to provide, you know, very specific

[21:10] kind of numbers around that. You can do the same thing with metadata. You can do the same thing with a whole bunch of different tasks. But we're all going to have to, as historians, get used to the idea that we're going to need to start to benchmark these models ourselves. come up with tasks where we can we know the answers already and we can very quickly give the models that task to test and see how well they're doing which enables us to gauge how whether or not we can trust them obviously if you have a ground truth an output that you know is correct and a model is not able to replicate that output you can't trust it on that task knowing that is important everything I'm going to say then is going to assume that this is something Right. And some of this is, you know, my built in assumption about how we work as historians. It may not always hold true for everybody, but it's it's where I'm coming from. My assumption is that most people don't want to just dump material into an LLM and say, write my paper. I personally, as an historian, got into

[22:11] history because I enjoy writing. It can be difficult at times, but I learned to think about a subject by writing about it. It's not something I want to lose in this process of kind of using a large language model. What I find interesting about large language models are the capabilities that they open up to us to do things that would be difficult or impossible to do otherwise. So in our current project, as an example, one of the things that we're doing is tracking fur trade families through, you know, voluminous records, right? We're talking thousands and thousands of Hudson's Bay Company fur trade post journals over several hundred years, Northwest Company records. records that are scattered across Canada, the United States, Great Britain, and they're all handwritten. And the problem is you don't know where people are. You might know that they exist from a marriage record in Lower Canada or from a ledger from York Factory on Hudson's Bay, but you don't know where those people exist at the rest of the time. And it's literally hit and miss trying to find them. Large language models allow you to kind of do that in a really effective way.

[23:15] And in our case, one of the things that we're doing, and Dr. Leddy's an Indigenous historian, and what we're trying to do is trace families that are made by Europeans and Indigenous people. And Indigenous names are often spelled very differently, phonetically, differently by French-speaking people than by English-speaking people. And so trying to trace people's names that are constantly changing in how they're represented is really difficult, but large language models are actually really good at being able to say, although these names appear different, they're in fact the same person. These models have the potential to do those types of things, right? To unlock large data. I think about what my career has looked like, and most of the things that I've done that I've found that are really interesting and noteworthy in terms of the things that I've found or published have come from basically finding unexpected things in unexpected places. I think that's true for an awful lot of historians. One of the things that I would suggest is that in

[24:17] looking at large amounts of data, the large amounts of data that have begun to accumulate as we capture more stuff in images and things like that, it gets harder and harder to go through that material. And the promise of large language models, also the peril, is that they're going to help us do that. And so that's my assumption here is that we're not trying to just automate history. We're trying to use large language models to do things that we wouldn't do otherwise. There are lots of examples of places, of course, where one wouldn't want to use a large language model for a whole bunch of different reasons, including that often they're not the most efficient means to find something out about the past. Sometimes though, they might be. What this involves to me is thinking about it in a very systematic way. And you'll see on the far side here, we've developed a website called Archive Pearl. It's a not-for-profit website run at Wilfrid Laurier University where you can go in and transcribe material. And what I'm going to show you today is going to be added in the next couple of weeks where you can also do metadata and some analysis as

[25:18] well with this. But the way we've come down to this is that if you want to use a large language model to do historical research, you have to approach it in a very systematic way, which means creating a methodology that you can use to establish trust with the material that you're looking at. That begins with processing material in layers. We conceive of this as being a source layer where you start with the raw images, you transcribe the material, and then you segment the material. I'll talk about those things in a second and then show you. You then have to verify that segmentation, transcription, and images. There are ways, again, to speed up that process and to make human verification is necessary at each one of these steps. You can then attach metadata to that material. This is like basically the steps you would go through to put anything in a database. Then you need to verify again. And then in the analysis stage, if you're using a large language model to do analysis, you have to be able to always verify what the large language model is

[26:20] telling you. And you need to be able to go back to the evidence, right? And so the idea here is to create a methodology that enables you to do this. It sounds really simple, but the reality is when you're working through this, you encounter a lot of different problems, which I'm then going to kind of describe quickly as we run through some of these. And then I'll give you a demo what this looks like on a website for how this actually works. So as an example here, we used to normally start with processing images. This is from Hudson's Bay Company Journal. It's a it's a page from Hudson Bay Company Journal. You used to have to start with processing images, pre -processing them for an extended period of time, at least in the old days, like six years ago, if you're using a program like Transkribus. The interesting thing about large language models is they don't need really high quality images. You can see that this is a fairly bad microfilm scan. It's been properly oriented, but they can deal with skewed images. They can basically read upside down and sideways images now almost as well as they can read text that's right way up for a human to read.

[27:23] But you can also use large language models to do some of that image processing. So you can now actually have the models do things like automatically rotate pages, they can crop them to the edges here, they can straighten them, they can do all of those things, if they're given the right tools, right? Image prep is a really kind of labor intensive process. It's not exactly a fun process to do. So, you know, it's nice to have a large language model that's capable of doing that. Transcription becomes kind of the main precursor. And the reason I say that is this. Up until this point, this is going to change soon, but right now, large language models are still much more efficient at using text than they are at images. You can give a large language model, if you strap it into the right harness, 300 pages, images to work with as raw JPEGs. Its accuracy and its ability to process that material accurately and reliably is going to be much less than if you actually give that material

[28:27] to it in terms of text. And at the same time, as humans wanting to verify what large language models do, it's much easier for us to verify material if we are able to search that material. And so transcribing material right now is really important. Transcription of handwriting has always been kind of a problem, even with OCR. Programs like Transkribus, which were revolutionary when they first appeared a few years ago, are able to transcribe text with about 20% word error rates unless you fine-tune the models, often 10,000 words or more than that, and oftentimes of training data to get the model to learn a specific hand. For projects like the ones I work on, we're dealing with documents that are created by hundreds and hundreds of different people, so it's not feasible to train a Transkribus model on every hand we're going to use. Large language models, though, and here I'm going to speak specifically about 18th

[29:30] and 19th century English language texts. I want to use these numbers, but understand that what I'm saying for English is true today, but the models are getting about this good or getting closer and closer to as good as they are in English on other languages. They're on the same trend line. They're just behind. We're able now to transcribe documents with about a 1.8% word error rate if you exclude errors of ambiguous capitalization and punctuation. We're all familiar with 18th century, 19th century handwriting. And sometimes the capital letter is clear. Other times it's not. What does that mean in practice? Well, humans score about the same or worse on this. The studies that we find show that people who have a lot of experience in transcription have a word error rate of around four to five percent, which includes, you know, any kind of a mistake, typo, things like that. the models are down to about 2%, a little bit less than that.

[30:31] We can actually reduce that even further. One of the techniques we've developed in our website is you can use multiple models, can read the same transcript, you can then compare them. And as you see here, you can highlight areas of potential difference. And so on a transcript where you have a 2% error rate, you can identify 80% of those remaining errors by having a human read only 5% of the text. And what that means effectively is on a 10,000 word text, a person has to read only about 500 words to capture 80% of the errors and reduce that error rate down to about 0.3%. And that's a form of human verification where you're going through the text, you're verifying things, making sure that it's correct. But instead of spending, you know, an hour on a page or 20 minutes on a page, depending on your speed and the length of the page, you're spending maybe a minute

[31:31] or two doing that. This is an essential part of the process, though. Getting the text right is really important. What we find at the end, you know, those remaining 30 words that are wrong in 10,000, those all tend to almost exclusively be modernizations of historical spelling errors, meaning of a document is rarely that important. If you're trying to create a published primary source, it becomes important. But again, this is where the use case actually matters, right? You can do this with tabular data too. Tabular data has always been a problem, even in OCR. One of the things that large language models are really good at doing though is writing code. And so you can actually take that same ability, and here's that page from that same ledger I showed minute ago, and you can have the models go through and autonomously figure out what columns they need to have and then generate the rows and break everything up in the way that you see demonstrated here. The error rates on tabular data now, if you

[32:35] again give the models the right tools and you strap it into the right harness, which is a coding problem and not a model problem, The error rates are slightly higher than they are with just straight up text, but they're about 95% accurate now and getting a little bit better. You can again go through that process of highlighting words and concentrating attention on likely errors and things like that. One of the interesting things about the newest models that are out there, like GPT 6.1 Sol or 6 Astra and Fable, the models from OpenAI and Anthropic, is that they can actually go verification now. I was recently playing on ledgers like this, and the models are actually able to now go through and find errors in digits by using the internal mathematics of the ledger itself. And it's interesting to watch them do this, right? Because what they're able to do is they first of all convert pounds, shillings, and pence to a decimalized number. They look at unit prices and quantities.

[33:38] They multiply them by each other. If they don't match up, they go back and numbers. And the reason that's important is if you remember that the models are predictive in that the things that they're, the way that they actually approach the world is through prediction of text, numbers are inherently hard to predict because whether something is zero pounds, 15 shillings, and two pennies is often completely unknowable from whether it's going to be zero pounds, 15 shillings, three pennies, right? You can't necessarily predict that unless you're able is, what the quantity price would be, what a probable number or a price might be in, you know, 1758 Albany. The models are able to do that now. So you can actually have them go back through, and this is what's pushing accuracy rates now increasingly up on tabular data, even above some cases what it is for textual material, and at least in some of our most recent runs, because the models are able to, again, use the internal logic of the document itself in order

[34:40] to produce The other thing to keep in mind, though, is that when you are working with documents, as humans, right, we go through a journal like this and we read the journal and we see the journal is a series of entries. we intuitively flip through that journal looking for the entry that we're interested in right or we read the whole thing but if we know something happened on september 10th 1773 we're going to go look up that entry to give that model that same ability as you're turning this into usable text you have to be able to put this text into a database in a much more kind of lower level of fidelity than we are, or a higher level of fidelity, a lower level kind of description, than we're used to in archives. In an archive, the diary kind of becomes the thing, maybe the page. Here, you need to give it to the model in terms of the actual day, because the model, when it later on goes to use this material, it too has to be able to

[35:44] say, hey, I need to look up something on September 12th, 1773. That means you've got a segment material. This is much harder to do than you would think because at one time a couple of years ago, page breaks where a diary entry carried over from one page onto the next. Models had a hard time figuring that out. Letters that would be interspersed sometimes with diary entries, tables, this all confused the models. Today, the best ones out there are able to just simply go through and very easily insert page breaks like you see here and know exactly where they are. Once you begin to segment that material, though, you then have a layer of material where you have an authenticated transcription that you verified, and you have it segmented so that the material is broken up. That's that source layer that we talked about. Once you do that, you can then begin to have models go through and actually generate the materials that you need or that they need in order to answer much more important historical questions later. That means extracting metadata.

[36:47] And this used to be an archival process more than it was an historical process. We might go through and code things. Some people might do this for their own reasons. There was a whole field called named entity extraction or named entity recognition, which, you know, has like mathematics been kind of blindsided by what's happened with LLMs. If you looked at how you used to try and extract the names of people from records and how you would test that in the past. A lot of the material that people would test on was based on pulling out OCR -errored text, so text that looks like there's problems with it, the typical OCR errors you might see from something done like a newspaper in 2008, and you try and pull that material out and categorize that as an entity to be recognized. Today, large language models are very good of being able to go out and pull out individual pieces of text. And this is done again through prompting and things like that. But they can extract the names of people, they can extract places, they can extract

[37:50] dates, they can assign keywords, they can tell you whether a document's relevant or not, all of these types of things. In our tests, the biggest problems we've had on this in terms of accuracy is with the latest models finding examples where they are in fact inaccurate. So when we do our testing on our benchmarks that we're building for metadata, one of the problems we run into is that the models make so few mistakes now that figuring out what the error rate is, is difficult. We will see mistakes made, but those mistakes are very often the same ones humans would make. And they're less mistakes than they are interpretive. So that if you were trying to, you know, extract the name of a place, are you using the historical spelling of the place or the modern spelling? What if that spelling varies within the diary itself? Should that be standardized from one entry to another? These are all questions that we deal with as humans using these records, and the models have to follow the set of rules we give them. The difference is we don't often articulate those things to quite the level to

[38:53] each other as humans as the models do, or as the models need them to be articulated for their purposes. But what we're basically finding is they are better than 99% accurate at doing this. And the types of errors that we see are generally errors of omission where they miss something. They might miss a name on a page rather than being errors where they are inventing things that just don't exist or something like that. That is exceedingly rare at this point. The other thing that's fascinating is the models are exceedingly adept now at actually finding implicit references to material. hear is that if you have a diary entry that says the men at work the same as yesterday, the models are able to figure out that that means that yesterday, the people are doing today what they were doing yesterday, which means you assign the same keywords you assigned yesterday to the diary entry for today. It also means saying that if you get a diary entry in December, which says that the

[39:53] man who was here in September came again, the model is able to figure out that that reference to the named individual from back in September. And you can imagine how this is important over massive amounts of data, right? Where what you can begin to do with this is you can begin to create a level of kind of coding and metadata fidelity that you couldn't do in the past with this material. All of that then leads up to analysis. I'm gonna switch over really quickly and show you what to do with this. What I'll say is when you're using an LLM, that have been kind of like systematically analyzed in this way and put into a format where they work with it best. The models are really, really good at being able to tell you whether or not a document is relevant to your query. They are not very good at telling you or they're not necessarily as reliable at telling you when something isn't relevant. And this often comes down to how we articulate relevance, right? So if you're trying to figure out, you know, go through this file and find me any

[40:56] papers that are relevant to subject X. How you articulate what relevance means is going to be deterministic for that model. They will still miss things, but they will also follow your directions. And increasingly following directions is both a blessing and a curse for these models. The other thing I would say is the biggest thing not to do with models is to not trust them to find negatives. So if you want to, for example, say, is there anything in this file that's related to my research project? And you give it a description you're happy with. If the model comes back and says, no, there's nothing in this file that's relevant, the problem is that today that is the most likely failure case. It's still unlikely. But the reality is if a model is going to fail, it is almost always going to be in missing something. And this becomes important in anything quantifiable or anything where, you know, a null answer becomes important.

[41:58] Getting positive results where you're like, you ask the model for, you know, can you find an example in this massive data of this thing that I'm looking for? And it gives you an example. That's something that you can then go and verify and use. If it says, well, there's nothing in there that is an example of the thing that you're looking for, that is something that you can't really trust the models to do yet. because, you know, again, how would you verify that without replicating exactly what the model has already done? All right, let me very quickly bring up the example here to show you what this kind of looks like in practice here. Hopefully everybody can kind of see that on the screen. This is an example of what the newer kind of version of the website looks like. This is a version that we built for our own project relations over water, which deals with the fur trade records I was talking about a minute ago. But the interface for Archive Pearl basically looks the same. And this is the updated version of that that we're releasing soon.

[43:00] Like a lot of software, you upload images. You can just simply drag and drop, you know, pages onto the screen. You can do that as PDFs. You can do that as individual images. You upload them as you kind of go. So here's an example of a diary. Not great or not all that easy to read. You can kind of see there's all this marginalia and stuff written in there. So we can transcribe that, though, using various models. And what we do here is I'm just for speed. I'm going to put it down to one model. We're going to use one model here. We're going to use different models to transcribe. but we will do i'm just setting it up this way so what i'm doing here is i'm setting up a transcription where we go through and we have the model basically do the transcription for us and um when we do this uh it it goes off it works in a pipeline and it comes back usually within a couple of of

[44:04] moments um takes a second to do and what you get back in this process the length of time it takes to process one document is usually about the same as it takes to process multiple documents. As we're kind of waiting for that one there, I'll show you an example of what tabular data looks like. So this is an example here of that ledger that I showed you before. When you transcribe that, it's able to go through and create an editable spreadsheet for you that you can go through and edit and use in that sort of way. Our diary example, while I was showing you that has already come back. So here's then the description. The way we've got this set up so that you can kind of see it is I'm using a specialized preset we have where I wanted to insert the proper data in there that's missing in square brackets. It does that. I'm going to zoom in a little bit because I know it might be hard to kind of see what this looks like. But we've also instructed it to write all the marginalia as marginalia noted beside the entry that it's relevant to.

[45:07] You can see it's been put inside square brackets over here. And you end up getting, you know, a transcribed version of the text. What you can then do with this is you can segment that text. And again, it's just a matter of hitting a button like this, right? What's happening behind the scenes as you're watching this is the API version. So this is the model that exists behind the scenes, behind things like chat gbt in this case it's gemini that's doing this is being sent uh the text it's being sent an instruction in this case the instruction looks like this it's a very detailed kind of like description about how i would like the model to segment the document how to detect boundaries uh what to divide one thing from another i basically lay out the methodology right and it sent this as an instruction you can send hundreds of these requests at the same time so they happen all in parallel. If I sent 100 pages, this would all happen. And that took about seven seconds to do. 100 will come back in seven seconds simultaneously.

[46:09] And it's inserted page breaks, as you can see here. Again, the principle is that this should all be verifiable by a human, you should be able to go through and edit those page breaks to reinsert them. Once you've got the document, those segment, as I showed you before, you can create a context layer like you see here. The context layer basically means you've now divided that single page into two different documents. And you can see over here that you've got the image associated with both of them, but then here are your different transcriptions. So you're now cutting up the diary into individual sections. And now you can go through and you can do things like extract basic metadata. We'll do this here. And this is again that same process. There's a prompt, which is hidden behind the scenes with a bunch of settings. And I've given it a defined list of keywords that it has to use, rules to follow in

[47:10] assigning those keywords, all the instructions that I want it to do. The way I approach writing these instructions is I approach it in the same way as if I was hiring a human research assistant to do this. how you would describe to a human who you could only interact with remotely over email to do this task. That's essentially how this works. And if you're wondering about whether we're simply replacing human labor with machine labor, the reality is we're hiring more in our project, more research assistants to work with the data than we are to just do transcription stuff. They're doing far more, I think, historical things than they used to be doing in the past. Here we get summaries. Keywords have been assigned. You get journal entries. You get dates. Again, it's all verifiable that you can go back and forth. You have citations you can kind of grab from the material. There's different ways to handle the metadata. You can imagine that in diaries, you might want to verify that the dates are all sequential. There's ways to do that using aware metadata where you give the material to the

[48:13] LLM to process so that it can flip back and forth and see different pages at the same time. Go look things up. edit its errors. Increasingly, we're using agentic processing to do a lot of this as well, where you can go through and have agents do this using tools, and it's actually a lot more effective. There's interesting things you can do with visual images too using these models, right? In this case, we take a map of Lake Huron, and I asked the model, in this case, GPT 6.1 Sol, to place a pin at the heads of all the rivers that flow into Lake Huron in the map, and it goes through and it places the pin on the image for us you can then go into the context layer there and you can go to the geographic um uh setup that we've got here there's our map again but you can also have a model go through and remap all of those historical points onto the modern points on the modern map and you can imagine that a lot of that requires um a lot of historical analysis because, you know, where those places are

[49:14] plate or where those points are on the map were often inaccurate in the historical map and are different named rivers and stuff today. They're highly accurate. So if you ask the models to place, you know, the pin at kind of where that river enters Lake Huron, well, that's exactly where the Serpent River enters into Lake Huron. You know, if we look at a different one here, this is the Mississauga River. You can see it's exactly where the river enters. And the models are getting increasingly better at doing these types of things. I'm going to switch back over to my presentation here just to kind of end off on our couple of rules, and we'll have some time for questions, hopefully. What I would say is, from my experience using this, we need to establish some basic rules for using AI in historical research. The first is you have to have a clear methodology you can explain and defend. What I'm seeing increasingly in journal articles that are coming across my desk for review are situations where people have used ChatGPT or used AI to do something, but it's very clear that they don't know how the model did the

[50:18] thing that it was doing. And what you need to be able to do is to articulate not necessarily what the model specifically did, how it arrived at its conclusions, but to be able to defend what the model did or how you use the AI. So in the case of that map that I showed you a second ago, the pin placement, if it's correct, is fine. But I have to be able to show that I was able to verify that pin placement, right? That if you are using something, a result that a model gave you, you have to be able to explain how you were able to verify that that result is correct. You can't just simply say, well, the AI has told me that it's correct and it assures me that, you know, it is 95% accurate that it's right because the models can't do that. This is not how they work. So you need to have a clear methodology where you can explain, this is what I did. This is how I was able to verify the results. And that often means thinking about your project carefully, because in some cases, as magical as the end result might appear to be, if you can't verify it, it's kind

[51:18] of meaningless and pointless. And there are lots of tasks where the problem of verification may actually take a lot more work than just doing it yourself manually without an AI model. And what I would argue is that in those cases, it's much better to just do it manually, right? The LLMs are good for some things. They're not good for other things. And we have to get used to kind of thinking about it that way. All your findings have to be repeatable and verifiable. One of the best ways to verify AI results, at least for your own kind of sanity, is to run the same thing on multiple models or through multiple platforms. And if you get the same results, you can be reasonably sure that you're on the right track, right? you also have to be able to present your material in a way so that somebody else can replicate your results. This is much more like the scientific method approach to history than maybe many of us are used to. But if you're trying to say that, you know, you used a model to analyze a very large corpus of data and you reach the following conclusions, somebody else should be

[52:21] able to take that same data, give your same methodology to another model and come back with the exact that's a problem. And you're able to, in fact, very quickly replicate these findings yourself and verify them at the beginning. And oftentimes, a failure to replicate indicates a problem in how you're setting up the model to work that you need to work through before you kind of trust it to go forward. Like I said before, positive findings are easy to verify. Find me an example of X. Negative results are very difficult to verify. Is there anything related to Y in this set of If the answer is no, I personally would not trust that because that is the place where 5, 10, maybe 15 % of the time the model might be wrong for a whole bunch of reasons that have to do with your prompting, how you define the problem, or even just how the model's reading the material. But if you can't trust that negative result, you should just do that yourself, right? So if you're looking, though, for an example of something

[53:25] that you already know should be there, the models are very good at finding that, then you can go and verify whether or That is the example of the thing you're looking for. Again, if you can verify it, if it's repeatable, then it's probably useful. One of the most important rules, though, would be to don't trust an AI to do things that you wouldn't trust the best human research assistant to do. I've had a lot of research assistants in my career. I don't think I've ever published anything that they have done that I've instructed them to do or asked them to do without myself going through that material meticulously myself. because I always, as the last point is here, want to be in a position to take full responsibility for the material that I'm putting out under my own name. And so although a research assistant, you know, might get some training from doing that and it might speed up the process, I'm always going to be kind of cognizant of the fact that I have to take responsibility. So that's just kind of a good golden rule to use is don't have the models do things that you wouldn't trust a human research assistant to do.

[54:26] There are questions here about replacing human labor with models, of course. I would say from my experience that what you can do here is you can have models doing things that are effectively much more boilerplate work and have research assistants such as you're able to afford to hire them repurposed to do things that are actually better training for those people in as historians, you can actually get them to do more analysis, to think through these historical problems of how you actually prompt the model to get the stuff out you want, because it actually forces you to engage deeply with methodology, with articulating what we know and don't know about sources, all these types of things. What I'm finding is there's a lot more work for research assistants to do here than there was before. And that work is actually, I think, of much higher value for those students in terms of their own intellectual development and training. than say, you know, giving them a journal and say, go off and transcribe this for a month and come back to me, right? In the interest of time, because we have just a

[55:29] couple minutes left, I'm going to end it there and stop sharing my presentation. I think we've got a bunch of questions in the chat, and I'm going to try and go through those chats as well as pull up Discord on my phone. So I'm going to do one for one, kind of working up from top. Are there any questions in Discord? Not that I'm seeing in the Discord. So maybe that's just how I'm looking at it. I don't know. If there is something in Discord and I'm missing it, let me know, but I don't see it in there. Have I tested Gemini 4 Argon on RTL scripts versus 3.1? Right. So this person is saying that they're getting better results, I think, on Gemini 4 Argon, which is the model that Gemini or that DeepMind released two days ago. Weirdly, what I've found with Gemini 4 is it's as accurate on English language text as Gemini 3.1 Pro was.

[56:29] There's almost no difference in accuracy. It's the exact same. You have to put it on high settings for thinking versus on minimal settings. That's how it used to work on 3.1. What I am hearing, though, is that it is, in fact, better on other languages other than English. And what I suspect is that if you hook up a different harness to Gemini 4, you're probably actually going to find that it is going to be better if it has access to an entire document. So, for example, the transcription is going to get more accurate for page five if it can also read page one, two, three and four and six, seven, eight. If there's material there that's going to help it understand what's on five. That seems to be kind of my hot take on it at the moment. But, you know, I'm interested to see what just came out the other day. So I haven't had a huge amount of time yet. When it comes to trusting, I shouldn't the approach be to look at explainability rather than just testing if it gives the

[57:31] right or wrong answer to a question. Fully agree with that. Benchmarking this is a way of figuring out very quickly, you know, how accurate is the model on something where we know the answer. That is something where you can verify, you know, a general level of accuracy as a starting point. Because you might find, for example, that if you wanted to use a new model that came out or you wanted to use a cheaper open source model for transcription or metadata, it might be useful to know right at the very beginning that that model is, you know, has a 20% error rate, so isn't worth testing. I would then argue that once you have that kind of baseline data, you're going to want to then go through and use that verification process that I was talking about to actually verify that on your task, you can explain what the model did and why. It is a two-step process, right, in terms of being able to know these things. Ah, yes. See, when we do the self-hosted chat, I've got to kind of

[58:36] scroll through. Yeah, so 96% accuracy rate for transcriptions is good as well. It is slightly better than that for Gemini 3.1 Pro. It will have to do with the prompt that you use. And again, the harness, which is basically how you orchestrate sending material in the model and process what comes back. It's going to vary from one group to another. but, you know, if you're interested in how we arrived at that, we have an article in Historical Methods called Unlocking the Archive, which looks at what we did for that in particular, and there's open source material on that as well. Yeah, and this is what we're seeing too with transcriptions where they are checking their math. That's a new thing, and it was very freaky the first time that that happened, and I watched the models say back to me, wait a minute, the pounds, shillings, and pens don't balance with the unit price, the quantity. Let me go back and reassess that. I thought, oh no, this is not good for, well, it's good for historians. It's doing our research.

[59:38] It's frightening on other levels. Let's see what we have here. Yeah, they will, if the internal math is wrong, they will introduce that. We actually ended up finding huge numbers of actual errors, scribe errors in the originals that we were looking at. Yeah, so I'll address one other thing. Should you be feeding the companies these documents? I would argue often not. When we use these models the way we're using them here, we have agreements with the companies we're using, which are non, the companies agree not to use the data that we are sharing for training purposes. Are they potentially lying on that? Sure, that's a possibility. If so, then all the hospitals and universities and other corporations using those models under the same agreements are also going to be filing lawsuits about those sorts of things. So I would argue, yes, be highly aware of what that material, what

[1:00:42] the rights are you have to the material and how you're using it, and also what your agreements are and the restrictions are with the companies that you're working with. So thanks very much for attending. other questions, feel free to email me. My email address is mhumphreys at WLU, Wilfrid Laurier University, wlu.ca, and I'd be happy to address any of their questions. Again, thank you for attending, and I hope you have a great rest of your conference.