Abstract

Moulds describes a summer sprint that processed 17,500 printed Scottish Court of Session documents, about 230,000 pages, in seven weeks at a direct cost under $500, using open models on university computing. Each page was read first by Tesseract optical character recognition (OCR), which supplies page geometry, and then by a large language model (LLM) that corrects the text, followed by passes that mark document boundaries and cluster documents into cases. He sets out four commitments for keeping machine output subordinate to human description, tests the pipeline on the Bakers of Dundee case, and demonstrates a Model Context Protocol (MCP) server that lets an AI assistant query the catalog. The Q&A covers attribution, the legibility of the Solr index, the roughly 10,000 pending human decisions and image formats.

Chapters

Transcript

AI-generated transcript. It was produced from the recording by speech recognition and cleaned up with AI. It has not been edited by the speaker and may contain errors. The captions on YouTube are AI-generated in the same way. Each timestamp links to that moment in the video.

[0:11] If it suits everyone, I will get started. So, good afternoon. Thank you for attending here and especially everyone online. I'm Loren. I'm from the University of Virginia School of Law. I am an historian and a technologist, a digital historian, work primarily as an archivist. What you see here is a legal document, which is not much to look at. This is an information, a written legal document printed in Edinburgh in January 1803 for the bakers of Dundee, who were fighting their own town council over a milling monopoly. Essentially, the town's mills could not grind enough... No problem sized. Let me start again. The town's mills couldn't grind enough grain to feed the town, so the bakers were carting their grain to nearby cities, Perth, Cupar, and

[1:11] Arbroath, and were paying for it themselves. So for 223 years, an historian working on mill monopolies or burgh politics or the economics of bread would have been unlikely to have stumbled upon this document. And not because it's lost, but because it was hidden, bound in with other cases, uncatalogued and undescribed. This talk that I'm giving today is about what has changed in how we read large collections like the one where this comes from. And how LLMs or generative AI are part of that change. But today I want to start with talking about documents. The tools themselves, the machines, are the latest chapter in a much longer story about how historians work with evidence. And the interesting questions are as much about the archive as they are about the machines. This is one collection I want to focus on where this document is from, is the

[2:15] Scottish Court of Session papers. And I'm going to try and give you an account of what happened when we tried to read it at scale at two different moments in time. 10 years ago, we had a vision for doing so. Every case, person, and place described, linked, and findable. It was a beautiful dream, and I do think the right one, but the moment wasn't right, and the economics most certainly were not. So I think, though, now this moment potentially has arrived. There is a sea change in what we can do with a collection like this, finding stories hidden within these boxes and volumes, questions we could not ask before, taking on projects that were too expensive or too daunting just five or ten years ago. And what I'll show you today is the result of an experimental sprint I did this summer, processing this entire enormous corpus of documents and building cutting-edge tools for reading across all of it. So to start, let's talk about these papers.

[3:17] What is the section paper? A little background. And this, I think, is one of the neatest things for something from 1710, by an act of the court for the speedier dispatch of processes. Every case argued before the Court of Session, which is Scotland's highest civil court, they had to have its pleadings printed in full. So the petitions, the answers, the memorials, the informations, the proofs, the whole paper conversation of a lawsuit would be typeset and bound. If you look at this third image, Someone wrote not reported at the top. That's important because no digest ever carried this. And written margin, someone has written a note to themselves. And there's an appendix. When an advocate would have submitted evidence, the printer also set that in type two. So that's letters, maps, account books, diagrams, voter rolls, ships, logs, many of which survive nowhere else. And these are basically archives within an archive.

[4:20] The also really important for historians, who's in them? Everyone you see on this page ended up in court. They were underwriters fighting over a ship lost in Morant Bay, Jamaica. Heirs to a Scottish estate traced somewhere into the state of New York, not far from Albany. And James Graham described the petition as a Negro slave and a seaman freed from the press gang while the officer petitions to take him back. People appear here at the most consequential moments of their lives, This is inheritance, debt, their livelihoods, freedom, with names, places, and relationships, and often their very own words. And for three centuries, the finding systems said none of what these documents are about. Said Kames and Morison and faculty collections indexed them by case name and legal principle. Every finding system, let's state, every finding system is an act of archival authority. It decides who can be found, and by the same stroke, who cannot be.

[5:20] So, again, another key point, how were they made? A clerk, quite literally, would gather up whatever the court heard this that week, and it would carry it, or month, and carry it to the printer. It would be typeset and bound. So a bound volume, the way it appears in its original order and form in the archive, is a printer's convenience, not a case file. So pieces of three or 25 cases would be interleaved, would be out of order. No one ever bound these case papers together in one place. If you want to read a whole lawsuit, the petition and the answers, et cetera, you would have to hunt down each of those across different volumes and different shells and put it back together yourself or different archives. And the print changes under your feet. The long S that makes session look like sephion, custom ligatures, catchwords, printer's conventions. Reading these is reading a two-century moving target. quirk to get there. How did these get to UVA? We're not entirely certain, but after advocate Andrew Skene died, his collection

[6:24] went out for auction in Edinburgh. Somehow it found its way across the Atlantic to San Francisco, where a law librarian in the 1980s from UVA bought it, and then it sat for three decades. So, and I don't even think that's because no one thought that they mattered. I think Scottish historians have said that for decades, these are valuable. It's called the Scottish, the session paper is the most valuable unstudied source for Scottish history in existence. Everyone, most people would agree these are of great potential value. But what we didn't have, it was a way to read them anew efficiently. So at UVA, our collection is 58 linear feet, about 3,200 cases, 10,000 records between 1757 and 1834. It's one of the largest collections in North America, and that is a rounding error in the whole collection of session papers. At the Advocates Library, the Signet Library, and especially the University of Edinburgh, they hold more than 5,000 bound volumes.

[7:25] That's north of 250,000 records. Their collections into the modern era can be described in linear kilometers. So three centuries of ordinary and extraordinary life in the British Atlantic world, almost entirely undescribed. and no real map to find things in it. So this is kind of what scale feels like. When it's in your hand, the leather is cracking, the shelf number is inked into the spine, that ragged compressed edge is hundreds of leaves. Pleadings are from dozens of cases. They're sewn together by a printer two centuries ago. It's dense, it's fragile, it's opaque. And every time you open it, you wear it out just a little bit, So let's start with our project and us as historians here. So our work on this really ramped up in 2016. Jim Ambuske and Randy Flaherty are my colleagues at UVA at the time. We went to Edinburgh to find cool stuff to sort of justify this work from the

[8:29] American side. So we were working on forging a relationship with Edinburgh to digitize as much of this collection as possible. We were there to identify documents relating to the Americas. and our professional, well-trained PhD historian strategy was to guess at where stuff was. And what we were looking for was the Atlantic world. So we pulled volumes and we flipped through title pages as best we could looking for Virginia, Jamaica, America. And we found really wonderful things, things that may have not come to light for many, many centuries. But as we were walking out, Jim said, lamentably, I think, even, for everything that we found, what did we miss? We had no way to know. The researchers, we were researchers in one of the world's great law libraries, and we had no way to find related things across the collection or to stumble on what we wanted to find. So this isn't unloved. It's not beyond digitization, but it is really practically unreadable.

[9:31] So let me illuminate this in another way. This comes from Jim Research, research, he works on Scottish immigration to North America. In some of the printed volumes of or editions of letters he uses, footnotes say things like this. There are references to this man in the minute books of the court of session, which means he litigated and somewhere there are case papers about him. And one of those men turns up in a single document that we hold at UVA, who was sued over an unpaid bond in Jamaica. It tells us how long he was there, what he was doing, that he held enslaved people. None of that is in that surviving letter. And there almost certainly is a response to this filing in the collection, either at UVA or Edinburgh. We just can't get to it. So I understand, so we want to understand that gap precisely. We have evidence that a document exists, and no way through. This is an access gap, less than a preservation gap. So we tried to fix it. In 2018, we launched a digital platform, the Scottish Court

[10:33] of Session Digital Archive, which we call SCOS. Our conviction was that we could describe these papers with all these inherited taxonomies. We could bring in people, places, and organizations as records of their own. We could link places to their parishes with coordinates. Case records could be built by close reading for the unreported majority of these cases for what a digest never had for them. We even asked whether a system could tell that this Hendry Dundas here was that Hendry Dundas there. And I think our vision was right. I'd still defend it today. What we could not defend, as I said earlier, was the arithmetic. So let me just give you a concept of work. It cost us, in three years, we produced about 4,500 case documents that were fully described. 4,200 of them were linked to their cases. We had a registry of 4,800 people, a gazetteer of 2,200 places, a vocabulary of 662 Scott's law subjects like Thirlage, Element of Air. And these records were built by hand, which is miserable work, especially when

[11:37] you're facing tens of thousands of documents in our boxes and a quarter million across the Atlantic. So under the economics of 2018, the project stalled. Not for want of conviction, not for lack of expertise. It just really lacked a way to process and read at scale. And we didn't understand there. What we didn't understand then, though, is we had not failed. We had built a really vital foundation. So what's changed? I think three things. First, the cost of reading, broadly defined, has collapsed. So open weight models, which your university can run on its own computers or you can run on your own laptop, can read dense 18th century print, long S and all, for pennies a document. And even more powerful and capable commercial models can read manuscript with remarkable accuracy. We use these models, we use these LLMs as tools to read. Don't ask these models to write for us. I ask it to read and to say what it saw.

[12:38] Second, I think this is really the neat part and one of the things we'll show today. Discovery has become much more contextual. A document can be found about what it's about, not necessarily by the words that's in it. And third, getting back to our foundation, we already had expert description to check the machine against. So that's the unfinished catalog became our ground truth. What didn't change is the cost of judgment. So the reading now is cheap, the expert judgment less so. But we had a whole core corpus of things we could compare against. So that I think is really the heart of this talk. What's new isn't necessarily a new system of discovery imposed on the archive. It's a phase of reading the things historians have always done, which is now possible against a whole digitized corpus. And it pays out in two currencies. One is quantitative. cases involved Thirlage. Where did defamation suits cluster? This is the archives as data, collections as data. And it also, I think really the most importantly is it pays out in the qualitative.

[13:40] We can find this page, this deponent's words, this artifact with ink and marginalia. So the archive exists as text. And I think what we have defined as we have moved forward is that the deal is that the first never circulates without the second. That if you're doing quantitative work on our collection, if you're looking for clustering or anything like that, every time results come out, you can trace it back to the original document. Every pattern is walked back to a case, into a document, to individual pages. That is our discipline's price of admission. Okay, again, so what makes reading at scale possible? Again, We talked about how we are asking models to read for what it means in context, not just for the words that are on the page. Search, of course, is always then the simplest way to see it. Keyword searches match strings. But if you don't know the 18th century term of art, if you don't know exactly what you're looking for, you may not be able to find your

[14:43] document. Now every page as well is read for meaning and passages that mean similar things cluster together. Whenever words they use. so in plain English we can ask it a debtor surrenders his whole estate to creditors to be freed from prison so you might get stuff aback about cessio bonorum or sequestration which is Scott's processes for these types of cases perhaps a Scottish lawyer might know to use those terms but perhaps as well a historian may not know how to use that or another researcher may not know how to use those terms and it works the other way too Words mean different things in different contexts. So you might have service maybe about a servant's job and unpaid wages, as well as serve or service might mean a formal legal step to inheriting land. Reading in context can tell us the difference between those two things because it looks at the words as well around them. So if you imagine your own corpus, which 5

[15:44] ,000 documents, 200 documents more than you can read closely well, with words maybe you don't know yet, with concepts you're not familiar with or unsure even are there, you can still interrogate and land on the passages that answer your question. You can still read pages individually as well, but now it's easier to find which pages to start with and which pages come next. So this summer I ran an experiment. In June, we had our corpus at UVA, which we had digitized about 35% of it, and a much larger section that had been digitized Edinburgh. We had these collections, we had these papers in the digitized versions of these papers, excuse me, in two different places, and all the human error over time that goes into creating an archive like this. So I started in June. First, we had to learn how to read it. It's about 12 days and about three different approaches to reading 18th century print before one held up. Then I let it read it around the clock for weeks with open models on our

[16:47] university's high performance cluster and then we could start asking it questions so remember three years by hand was 4,500 documents in seven weeks we processed 17,500 documents which was about some 230,000 pages readable and searchable it's different kind of work that's not the same kind of work as we were doing but it It is similar and it changes what's possible. Our direct cost was under $500. And it's important to be honest about what that number means. This was done on research computing at the university. That's not free. We pay for those machines, but that is shared infrastructure that's available to many of us and our colleagues. It's the same clusters that engineering uses or astronomy, health sciences. And if you don't have access to it, there are affordable alternatives to do things, smaller models that use less computing powder. They may take longer time. But our real bottleneck in this one was not money per

[17:47] se. It wasn't even really human time. It was sharing time on the clusters. Okay. For those of you who are just getting started, I want just to mention this too. This is done by one person who knew this collection really well. As an archivist, how it was made, how it was digitized, why this collection would matter and to whom it might matter, and someone, myself, who could get access to university computing. We didn't really have an API budget. We didn't have a vendor contract, no dedicated grant. I do want to state, though, that the research component of this or the experimental part of it, as well as some of the tools, has come from a partnership that I am part of with UNC. It's Mellon-funded for us to come up with ideas, shared standards, and experimentation, like this. Essentially though, and actually very clearly I want to state that none of the data in the corpus came from a paid frontier model. The processing itself ran on open models on university computing, and we didn't

[18:50] have to wait for a grant cycle. So what we didn't really need money, what we needed was access and a deep knowledge of the collection and a set of goals or questions to ask about that collection. And so one of those, or at least one of those parts, is probably sitting on your desk or in your local archive right now. A little final thought about framework and method as opposed to a product. This is a framework, and I've even, Jaresh this morning, showed a framework that's very similar to it. This isn't something you can just download and apply stuff to. I've used it on numerous collections, two in particular. The one on the left is a 20th century federal index card series, which is relatively regular. The information appears in the same place. Once you understand the variance on it and where things should appear on the card, you can do this over millions. It's really a horsepower question. The session papers are different.

[19:51] Every volume is different. And that's really just the outside. You can see on the right, we have different shapes. I don't know why that image is rotated. Sorry, guys. and the print conventions drift document types vary unrelated cases are interleaved as we've said before and there isn't really a one-size-all fits tool but what transfers between collections or corpora is this method with places where knowledge of your own collections plug in and so we've seen a few of these early already today we read every page twice um robin earlier was talking about OCR and LLMs. I think this is a different way of approaching this. We did everything in OCR first, in Tesseract, and then asked a pretty large trillion-perimeter model to check it and correct it, and also get page furniture, which is page number and marginalia and all that sort of thing. So what's important about that is the Tesseract knows... Tesseract, or an OCR engine, can give us geometry of the page, where the words

[20:53] appear on it. are terrible at that. But when you combine those two processes, the OCR, the ALTO, the geometries help it along and help it in context of the page, and it can correct for those ligatures, the long S's, the fragmentation and issues with the text. So, the old tool knows where things are on the page and the new things, the new models, the new method knows what's on the page. use the frontier models reading as the benchmark. This is, we used Claude and we used OpenAI to test. And then we held open, and then we tested against open models who we could afford to run at scale until they were giving us the results that we wanted. So then we described it first where one document ends and next begins because many of these were just straight through volumes, straight scanned without any breaking up at all. Then we did six kinds of description, what kind of paper it is, who's in it, which place and in what role, what it's about, a short summary in our cataloger's house

[21:56] style, and what's inside of it. Then we gather the documents together. Basically, the important part for this collection is taking individual documents and binding them to individual cases. Then we have one door where everything enters the catalog through one audited path. And then we also have one search core with several ways of accessing it. The website, a map, a network of people, and an AI connection. Live shortly so these steps are a template or a guide I think they can work on many different collections What makes them work on this one is the knowledge our catalogers were able to apply to it So just so we go through our four commitments here I Said that what fits the model to a collection includes the rules about what a machine is allowed to decide on its own that is really, really important. It's what shaped the commitments we made. Four of them have governed every stage. One is that human description is an anchor.

[23:00] It's the machine is always measured against the human catalog and it never overwrites a value a person entered. Two, autonomous where it's safe, human where it's not, and the line is drawn by the kind of decision, not by how confident the model sounds. Three, one door into the catalog. with which pipeline, which version, what infrastructure, what evidence, and is reversible. And then finally, interpretation. We always prioritize interpretation and never allow silent assertion. So machine work is always labeled as machine work and the page itself always appears right beneath it or near it. So a scholar can disagree with what our pipelines have produced without losing the record. That's what I would argue makes it trustworthy enough for it to hand to you. all right so let's look at a document it's really quick the one that i showed you in the the first page the bakers of dundee we're back to our day or bakers uh um it's important that we sort of that this is one of our early documents that we

[24:05] process by hand by humans this is one of our gold cases that we set the standard by um i think that's what made it a really good test for us to talk about today we ran our full framework our pipelines on it um as if no one ever described showing what our catalogers wrote, and then we would set those results side by side. Every piece of evidence it cites comes from the page. I will show you that later. From the page alone, it located the filing date, docket, the parties, and the printer's imprint. Then it figured out it's a what, what it's an information. It was argued by George Joseph Bell, resolved to his record in the registry. Then about the who, Baker's Incorporation and Town those were new, those were questionable, so those got paused for the curatorial queue, so we had to confirm those before they went in. Then the third, what is this about? Thirlage, that's a mill monopoly servitude with 31 supporting passages, which gives us a first pass at legal historical reading, and because we had the human reading,

[25:10] we could see how good a pass that was, and that helped us refine the approach for the much larger collection. Okay, and just really quickly about the places. Putting a name on a map is relatively straightforward. Geographers have had good tools for this for a long time. I think it's cool to mention that the hard part that used to take a historian with a document open is knowing why a place is in that case, in that document. So the reader we asked to find the 57 mentions of places within this document worked out why they were And so we found out that Dundee is the disputed ground. Perth and Cupar and Arbroath are where the grain is going, the baker's extraordinary expense. But Haddington is there as well because it's a precedent, the lawyer's site in the case. So again, every judgment carries its passage in page, so you can check it. And we multiply that by thousands of cases and you start to get a geography of litigation. All right, so if you open the Dundee page today, which we'll see,

[26:14] on its description, that's deliberate. Our catalogers, remember, it's human value. It doesn't get overwritten. Our catalogers wrote it. And the machine won't overwrite it. So however, the machine's work appears on thousands of documents in our catalog, and our catalogers have never reached. So I've sort of clipped one there. That's from an 1825 state of the process. There's the summary. You can see it below it. And then there's the label that's AI generated. And if you hover over it, it tells you which model wrote it and when. click on it, it leads to a documentation page, and you get the model, the prompt, and the date. And every machine written field carries one of these. And the better model next year, if we add that over top of it, rather than erasing that one, carries a version note on it. That's the obligation that comes with reading at scale, that we must be able to show our work. The page says this is evident, and the model thinks the page is about this, those two things must never blur. okay so how do you trust something like this at scale and not by assuming it works and we've talked about how we would

[27:17] we had this sort of iterative process of taking one document that is our gold standard and testing our models and our pipelines across it um and then we spot checked and to reach at full scale sometimes we have to say that this is good enough It's not perfect. It's good enough to be useful and open to testing and revisiting. What matters is opening entry points into the collection, then handing it to researchers to explore that and synthesize. So every researcher who finds one of our mistakes makes it better. And then finding these gaps between the collections also means finding the gaps we've created ourselves. Okay, let me show you some stuff. our site you can visit now it's at scos.law.virginia.edu let me make sure my internet is working here okay so

[28:19] this is our site right this is a case document it's an information from 1784 this is just the one I had up I'm on the wrong page there we go you can see here the tags I'm talking about the abstract was AI generated the subjects the place is mentioned have a badge on them when you go over them they highlight up if you click shows you that for people's subjects and places were individual pipelines that used Qwen3-VL, full weight, right? 8B, no, sorry. So we used a highly quantized one, sorry. But you can see then if we go through here, the very simple prompt that we used, where we ran it on the HPC cluster, and then the prompt we used to process the data.

[29:22] This is pretty straightforward. You know, and we can also do something like, if we want old school, I can't really see that but there's our document bakers of Dundee we can take a look the other thing that we did and we mentioned earlier that we've created auto text or coordinate text in the OCR. We can use that to find all the instances of Thirlage in the text or strings as well. Jump to those pages. Pretty basic. Internet's pretty slow. Sorry, guys. Okay. Now the cool part. One of the things that we created here

[30:24] Got to find my cursor. Hold on. I'm on the wrong page. I'm so sorry. OK. So we built an MCP server to connect into our database, this model context protocol. It's basically an open standard. You think of it as a universal adapter data or tools. Ours gives the assistant a small set of tools to help search within this archive. It can search the whole corpus by keyword and by meaning. It can open a document's record, look up people and places, find similar cases, pull up an individual page. So when I ask a natural language question, the assistant doesn't answer from its memory. It doesn't try and invent anything it's going to first privilege asking our corpus querying our corpus um so let's take a look at that

[31:27] that's the cool bit and i've asked a couple of you today just at lunch and things about some queries you could ask um does anyone have anything that's of particular interest to them in their research that i could type in here and see what we can get while and while fable will crunch through it or we can do some canned ones i'm happy to do canned ones too I have a question. Sure. Were there any limitations in trying to travel to Scotland? I don't know. Great. Let's ask. Italy, you said? I couldn't tell you. So that's going to crunch its fable. It takes forever. It takes lots of stuff. But I'm going to show you a couple of the ones that I canned earlier today.

[32:28] Jack from London asked about his colleagues who were interested in slave narratives in English law, correct? Sure, good enough, right? So I asked it, and here's what we got. So it wants me to know that England is a different place. It's not Scotland, thank God, right? but that they have a different they have a different legal system to note that but then what it starts doing is like it talks about freedom suits, it started looking through and finding Joseph Knight and Wedderburn which was a very very important case in Scotland about about slavery, we can talk about that later as well but what's neat about this is you can immediately click in and see this document within our collection as well. I asked one of my favorite ones. Our smoke test for years was ask if we could find instances of kelp. But now we can ask why kelp important and how do we get to gunpowder production and

[33:33] the American war and all this sort of cool stuff. That is something we can ask through here. I've asked about transatlantic insurance fraud and I found this really cool case here that tells me all about it and some things that are related to it. One of my favorite things about this case is, about this collection, this corpus, is again, we have people who are deponents who are giving testimony in front of court. And sometimes those are people who almost never, ever would appear in the historical record, formerly or even by chance. But now we have women who are being, giving testimony in front of the case, in front of the court. And so what I've asked it really simply is, can you find any case documents that print I can't read it, sorry, I'm gonna come closer. That print, the depositions from female deponents where we have their verbatim words, I'm interested in how this project surfaces voices and stories that normally are absent from the historical record. And the first thing it says is this turned up a lot.

[34:34] And the pattern of where women's verbatim testimony survives itself is itself telling. In paternity, adultery, defamation, and marriage validity cases are the genres that put ordinary women on the record, midwives, land ladies, neighbors, and a few standouts. And then it starts listing those instances here throughout the collection. Some of these aren't even really fully processed cases yet that what we know we have partially processed through the pipeline. Let's see what people came up with about these Italians. Still thinking. But we're using this big monster of a model and asking it to look through the whole collection so these things will take four or five minutes. Yeah, go ahead. I have a question about the types of questions one can ask.

[35:37] So clearly this is very powerful when it comes to reading material that is listed verbatim within all of the documents. I'm assuming it synthesizes it as well. Yeah. What about classifying certain kinds of knowledge, for instance, the explicit knowledge that is obvious because it's stated, as opposed to extracting tacit knowledge that may not always be obvious or somehow implicit in the actual legal proceedings that are there, but maybe something that is contextual to these legal proceedings, but it's still a pattern. I don't know, some kind of reading practices of judges, some kind of social rituals, would this be also available to the query? So one of the things that I want to leave

[36:39] the room is this is something that we've built over the course of weeks and where the limitation we're at now, or at least the blocker right now, is we haven't, I haven't used this or haven't stress tested against historians actually using it. So one of the things that I can understand, oh, this is what we, when we were developing this project, these are the things we've stumbled upon. Here's who I know is interested in this material would want to ask questions about. But it's a really, really, I think, important and possibly even really generative suggestion is that we integrate that into it now with the agents that are, this is just in Anthropic, but it's an MCP that we could connect to anything else. So I think it's a really interesting question. And sort of the point we're at now is we leave this experimental phase that actually kind of produced a product that might be cool, that what do we have to do is start thinking about this as a platform for accessing this data in innovative ways, that maybe that would be something that would be really cool. Yeah, go ahead. I'm just trying to wrap my mind around like the MCP part of this. Sure. I have not worked with them before. is it that you are hyping the context

[37:43] from the archive and everything that you have digitized that's done through this like first pass with like a Qwen model for example how are you routing that into another platform like plot like so so underlying all this is a database right and then it's it's next to that is a Solr database as well that we've used to index the text and within In those fields within Solr, we've also clustered, we've embedded, okay, we've said, what is this stuff about and what's nearby into it in this sort of vectored space? And it's really complicated stuff, but basically that allows us to say these things are related conceptually. And so all we've done for an MCP is create a series of tools that instructs it that the type of searches that will be productive for accessing the database and getting information out. And so when we say we're looking for this type of stuff, it will preform. The agent should write up a series of very simplistic queries that will start getting data. It'll parse those, read them, see what it's getting, and then iterate through and reason through that sort of process of thinking, if you want to call it

[38:47] that. Anyone who's an expert in MCPs can correct me, but that was my big layman's guess. Okay, thanks, dear. Well, I was just going to suggest that you expand the thinking there. I hope some show kind of like what... Sure, sure. Yeah, that's super helpful. What's neat about this is it's not restricted to my platform, that this is open to whatever harness or agent that you're trying to use, and you can see what it's thinking. You can watch its reasoning. You can interrogate it. You can have a conversation with it. You can ask it to parse the data. You can ask it to prep the data. You can ask it to export it. that I, you know, I haven't put this to the test. I've used it on my own stuff, but I've not used it for my own research because I'm not a Scottish historian. But I do think there's moments for this to be a model that we apply to other large corpuses or small. Yeah. The question I have actually goes directly, I think, to the fact that you're an archivist at a law school. It seems to me that you've created

[39:51] a method that has a kind of, say an intellectual property dimension to it. Are there concerns about the kind of integrating this into something like Claude or something else that, you know, potentially cuts against or something that can be essentially taken or acquired or utilized by someone else in a way that doesn't allow for attribution? You mentioned a lot of use of a lot of open systems and I'm a big fan of that for a lot of reasons but I'm wondering about now you know Claude is a closed system and I don't understand this whole thing about MCP too I don't know what it stands for to be honest that was my question what does it model context protocol it's just a it's just a for it's a way of formatting tools to the scripts that go in yeah we can talk about so let me let me let me say I'm going to answer this question and then I have two or three more slides just to sum it up and then I'll open it up for more further

[40:51] questions but I think that super duper vital when we think about what we do as archivists because this really does represent two different very important values. One is this is downstream labor of me as an archivist and that takes a lot of work. It takes a lot of expertise. It's been a career of mine and I use the word career there because my whole career up to this point, 15 years as an archivist, has been How can I make my stuff available to everybody? Use it however you want. Here's my license. Just attribute it to us. I know for a fact that my stuff is used in these models. I cherish these tools. They have changed how I do work. But I do know that my stuff is being used without attribution, atomized, decontextualized, which is an anathema to what I do. So I will say it's worth noting that at UVA, we are dean of libraries, of people is pushing a new protocol to say let's pause on putting stuff on the

[41:53] internet and making it accessible to be hoovered up i think it's a worthwhile thing to think about why i think this collection is accessible is acceptable in that work is because it already existed on the internet for the last 15 years 10 years right so i'm not adding to this we want to but we don't have immediate plans to do so because we don't have a path forward in light of this question but I think it's a wonderful question and it is really important for us to understand as historians as archivists people who possess really unique information and can package it in a way that makes it accessible for humanity to know our stories collectively to have that use let's not I say stolen, but used without attribution is troublesome. So super, super important question. Let me just finish up because I have just a few slides more, and then we'll try and answer a few more questions. What app am I using? Oh, yeah.

[43:00] Okay. so i i do think in light of that conversation one thing that is worth highlighting it's early on when we were working straight from ocr and keyword search enslaved people were actually very difficult to find in this collection um searching for the word slave or other obvious terms would surface only a handful of documents scattered sporadically through the collection but having looked at many of these and knowing the history of this period in the court we knew we'd read enough of these papers and we knew enough about the economics of that time history of the British Atlantic that we knew that their lives and stories were documented within these we just couldn't reach them so now after this new sprint the whole corpus fully newly processed just on cursory searches we found 1247 passages that are normalized for language so there are some things that says Negro Sally which is actually a pub but we've started figuring out that we have 1

[44:03] ,247 new passages in about 108 documents and counting across all of these pages. And I mentioned Knight v. Wedderburn, the case that ended the enforcement of slavery in Scotland, in the colonies, and one of the most famous cases in Scots law. It's not a unique document that sits elsewhere, but it is just something that's a really important document just sitting in our collection. learned about a man named Lachimo who put forward as a witness in 1770. So a handful I guess of these were against more than 1200. It's a huge change. And the silence wasn't necessarily one that was in the archive. It was one in how we were able to look.

[45:03] The old indexes were built by lawyers in a commercial imperial society, and they recorded what mattered to them, the legal principle, the estate, the debt. Reading every page doesn't erase that bias, but it stops the index from deciding in advance who counts. Reading at scale found these people, their voices, and their stories. And that's what this type of you and it's no longer only institutions that can afford to read everything. I do think that that's true. Okay, so finally, what about you guys? What about all of you? I don't think many of you, most of you are sitting on 230,000 pages of documents and a full cataloging department, but many of you might have a project of a couple thousand petitions, a box of letters, a box of photographs from a research trip, which is the way to do it now. And what I want to leave And now that is doable at full scale by you, not taking samples of it, all of it, doing all of it. And it doesn't necessarily take a big grant or a lot of funding.

[46:05] And it doesn't take a team of technologists to do it. It takes a corpus and a method and a knowledge you already have, the terms, the people, the places, a few dozen documents you've read closely and suggest further exploration. So we start small, check it against your own reading, and then we run everything. and then you can bring it those framing questions that we were talking about, right? Or the methods that you want to introduce to that data. You, the historian who wants to ring that evidence out and the stories out of these documents to understand the decisions people made in the past and the circumstances of their lives. So the processing and reading that once needed a funded team in five years is now a vastly shortened time period and far more affordable. okay so i want to let me widen the lens here because i want what i want you to take out of the room isn't necessarily a pipeline it's more of a posture um so i was recently talking i'm not i don't mean to name drop but i will i was recently talking to Ed Ayers um who asked like how do we find

[47:08] the forest when there are so many damned trees and ed observed that ai has suddenly handed us and everyone else, you know, chainsaws and bulldozers to clear them, which is enormous power arriving wherever we ask for it or not. And when Ed and I talked it over, we landed in a very similar place. We don't want to use AI to clear the forest. We don't want to skip through things. What we're using it for is to try and map those trees so historians can decide where to walk. So everything you've seen prepares the archive for transcribing, structuring, tagging, the answering questions within the pages, the historical reason, what a pattern means, whether an anomaly matters, where to walk, that all that work stays human. So these tools work best as collaborators and they work worst as authorities. And this system is built so that they never get to be authorities. And this, I don't think as some people would call, I don't think that this is rupture. I, and what historians do, I think we've

[48:09] microfilm and databases, and text encoding and full text searches all expanded the scale of what we could see and none replaced interpretation. So what's new, I think, is a sea change in how much of the record we can bring our craft to. So Ed, in a recent talk, ended it with what history is already good at, which is surprise and anomaly and rupture. And I would add empathy to that. AI can't produce those things. it can help us reach them though by revealing evidence where it's hiding and the bigger the record gets the more those human capacities that we have are worth so that's my talk so the i guess the bakers of dundee waited 223 years to be findable they're findable now and So are those midwives, and so is Latchimo. And I'm opening this up to any questions. Just to remind you to make sure I'm thanking it, I want to make sure that we

[49:13] mention UNC's partnership. They've been really remarkable in the Mellon Foundation. This was helpful because of a data analytics grant at UVA, as well, just to sort of get me connected into that system, as well as, of course, my UVA School of Law letting me take the time to do these sort of projects. So, yeah, we have questions? Yeah. This is fantastic. I have two questions, one on the history side and one on the archive side. So on the history side, there are a lot of moments of interpretation that go into the process that you described. Indexing concepts across 200,000 pages. You pointed out that words from 18th century Scotland could have different meanings and so forth. Did you account, did you historicize the language, the concepts in any kind of way so that when you sent the AI models to begin to index it, to

[50:17] read it, not to summarize it, to read it, that could influence the interpretation of what it was reading? So, no, except for one major exception. The question is, did we tweak our input to help the AI agents or our tools for our entry points to help make sure that they understood better than they would out of the box? To turn them into 18th century Scottish historians while they're doing the work that you're sending them to. So no. And in fact, it was really just us taking the text and teaching it what was wrong with its current understanding of what a printed text should look like. And then it's going to see the things it's going to get hung up on in terms of spelling, ligatures, and that sort of thing. That is really the only time we interjected except for one time when we did.

[51:18] in the MCP in particular and some of our vocabularies, we tried to make sure that we were very clear about what words meant for enslavement and what didn't and tried to look through them and clarify to the AI what that meant. That was it. So this is really kind of straight from that processing pipeline with very few thumbs on the scale to say. Gotcha. So to speak. I think more and more archives are going to see a pipeline like this. It might not be exactly the way you all approached it, but what advice would you have in terms of what may seem familiar in terms of the kinds of practices and methodologies and what would be completely different that they would need to do, say in preparing their collections for MCP access or indexing it for AI model access and so on?

[52:21] What needs to change? So I think the first part is if you have lots of data that's well-structured that is from archival collections, it's not hard to prepare that programmatically to be available for indexing and searching. I don't think that that's a new thing. The only thing that is cool about the MCP is it makes it far more efficient, and it gives it a structured set of tools for interfacing with it. And so it's not particularly fancy. It's just kind of a new, cool way that allows us to, instead of copy-pasting text from one place to other, copying pictures of images into an AI agent, this is one that directly allows it to pipe into the data we've formed from the corpus. And so for advice to archives, to how can we, how can an archive do? I don't have much advice to archives. I constantly try and follow where the archives and archivists and SAA and all the cool digital archivists and digital historians have done work to sort of prep us for how do we have great preservation standards, both in terms of

[53:22] physical and digital preservation. What is metadata? How do we get people to interface with it? What are the standards for metadata and the way we frame things and phrase things? So I don't have much advice to archives. I just thank all the archivists who have helped inform me in my career. That's what I would say. The one thing I just want to remark that this sounds so cool and it sounds so easy to do in seven weeks because we had all this stuff digitized from a much easier, earlier version of this. The big blocker now, the costs have come down in terms of processing, in terms of reading, in terms of creating cool tools for discovery. that is far cheaper. Digitization is still a pain in the ass. It is real expensive. And this collection is a case in point. Edinburgh spent so much more money on their collections because they were in such more shape than we did because ours were, before we got them, disbound. So they were ready to go. Boom, boom, boom, right? Edinburgh's are bound, old, cracking. It's cheaper if you have hyper-computing. Well.

[54:22] I mean, what about the small archives and libraries out there that would love to do Anyway, I'd love to talk about it. Great, cool, find me. Yes, yeah. Peter. Yeah. I mean, it's kind of like looking into the future of what I think most archives are going to adopt at some point. Cool, thank you. My question is kind of related to the one that was here, which is the Solr database that contains the embeddings. How open is that database and legible, I guess how legible is that database to humans who want to review the embeddings? And to what extent do you feel like you have or you could have an understanding of the connections that are being drawn inside that index? My sense is that it's literally encrypted. Well, no, I can see it. It's just in a Solr database. I mean, but it's pretty obtuse. I mean, but you can, I think the way I would respond to this is, again, this is

[55:23] something that, this is like bleeding edge stuff. I just made this. brand new version of it and a lot of that slide where I was talking about sometime we just have to have faith that this is good enough and let's see what we get I think that might be where we are right here and these questions about how do we how do we interrogate how these things were vectored how do we interrogate what is being surfaced by this particular system versus this one and what is the that those are wonderful questions and I think are actually just as generative as the historical documents that are in this but I'll tell you I have not done a lot data. I mean, Robin's talk earlier was all about the error rates and how do we make sure that this is good? How can we make sure that we have good recall? How do we have good precision? Mine is just sort of like, look, I did it. Cool. Now what? So I think that those are very different stages of the process and for different purposes. Mine was much more about how can we get legislative history, how can we get legal histories and the lives of people out of this hugely

[56:23] dense, massive collection? and then we start asking the questions about making it better. Maybe. So there's a Discord question I want to answer. It was from Mark from before the talk even. Kudos to you. I do think it's a different question. His question is, can small archives do this? Can small individual historians do this? And I would absolutely argue yes. There are many really incredibly powerful tools at our fingertips here that allow us to do this. It does take some hand-holding. It does take a little bit of bravery. I have lots of people, not lots, but numerous faculty members and researchers who come to my desk now and say, like, oh, I got this. Can I do something with it? And I'm like, yeah. And if we can walk through it, then in a couple of instances, they've gone and done it, right? And they didn't have access to the cluster. they were using effectively, they were using, you know.

[57:25] So Mark, it's a great question. And I'm blessed that I don't have to worry about that in this cases, but there are many of us where I think in the future, as these models become increasingly powerful, but also increasingly affordable and small, that we will be able to do cool stuff with them as the lone historian on their office desk. Yeah. For one's worth, and I wonder if somebody you're in I was recently looking for good examples of public databases that have APIs that are both doing public service and are kind of like protecting themselves in some ways. And I actually thought the National Archive was an outstanding example where like you have to have an API key so they can actually tell who's using it. Yep. What they're using, you're kind of like capped at how many queries you can do in a certain period so that allows you to kind of kick out people or identify people who are trying to kind of download the full data set. example that I don't know if anyone works there wants to come and see a lot of knowledge but I thought it was extraordinary kind of creating the public or maybe

[58:28] not extraordinary but like a great working example of someone trying to balance those two concerns yeah and it leads to me why this MCP isn't public yet is simply because this is on a cloud run on google on google cloud and so it takes the 60 seconds to warm up and then like I you know every time I use it it has very little bandwidth and we pay for the number of times I have to spin it up so like this isn't operationalized but it's something that and I have to access through an API key simply because I built that into it because I don't want someone without an API key just to know the endpoint and start hitting it because I don't know what I'm doing so I'm trying to make it work good for me and trying to eventually think about how we can make it good for everybody else. Any other questions? Yeah. Yeah. Yeah. Yeah. I didn't get much into that, but yeah. But from your experience with this, if you were to try and run this over your whole dataset, how much work would be held for curator?

[59:30] or if Edinburgh managed to digitize that whole archive, how much work do you think would be held for curation? Because that seems to be the ultimate bottom there. So the cheap answer is that depends on what you want. So like with this one, when I first started, I held a lot for curation and just to see what do we need, what do we want to prevent from being autonomously processed? And then when I looked at it, I was like, I don't want to do this. We can do this better. And so I opened it up just a little bit more processed another 10,000 in that way and fixed that sort of glitch. And so for most of those things, I don't think, at least from my perspective, that that diminished the value of the data. I don't think it did. There might be some of us who go to the other end of it, it's like, who cares? Get it out there. And then there must, like, where you have a real statistical research question you want to do, or it actually affects people's lives in very, very real ways, then you might want to have a really robust curatorial process that we have people humans looking and comparing and saying

[1:00:34] okay we might also crowdsource it if it's not that sensitive right and if the errors in binding or the errors in um finding bound page boundaries is not that big of a deal and people are going to help us sort of sift through these volumes because there's a little anomaly here or there that's fine right so my answer to you is like with ours where it is now probably about 10,000 human decisions that need to be made for adding new subject terms, for confirming that one person is in fact this person in another place because it's based on chronology and location and all that sort of stuff. Other things are like, why is this page single, in the middle of nowhere, but clearly part of a text? There's just all these things that we have to do. I would say it's about 10,000 human decisions over the millions of inference that we've done on this. The honest

[1:01:39] answer is like, I want to tell that all that human labor was super duper duper duper important. But like 4,500 is overkill, right? Because we had, we had the span of multiple decades. We had lots of different document types. We had lots of different printers methods and that sort of thing. We could account for that. So it's not necessary to have 10% of a collection done. It depends on what you want and what you have. And if you go through it and say, yeah, this is getting me the results I want, and then you iterate through there, that's fine. Otherwise, you might have to have a really robust schedule where you're having orchestrators compare across it and retesting and retesting, I don't know. But it really depends, again, as what you bring to it as the researcher, the technologist, the person who wants to extract and use and structure this data, what do you want? What kind of questions do you need to ask? And what type of information, what kind of data, what kind of stories or materials are you getting into it? Yeah. I was hoping maybe...

[1:02:40] I'm not repeating. ...more into process, because I'm very encouraged by the fact of the observation you made about That's approachable. This is cost-effective for people, individual historians. I routinely deal with document sets into the 12,000 documents myself in my work. And I can't count how many pages. So I'm curious in all of this. The reason why I'm here is to learn how I can leverage some of these technologies in my work to extract more around the research that's oftentimes many, many years, that's gone into this. So I'm wondering if there's some suggestions, some principles that some lessons learned about how you begin and how you proceed. So for instance, I asked this question. We were in the same panel with Robert. I don't know if you were still there. But when you digitized these, did you do it at TIFF? Did you do it at PDF? Did you do it at PNG? What was that like? And then what kinds of open models?

[1:03:41] You mentioned a few. I wasn't clocking all of them. But what are kinds of things that, if one was to undertake this on their own scale, because it seems approachable, what are some principles you learned or someone else consider? So, like, you know, the impulse to say, oh, we're in an archive, so we use preservation-quality images. Sure, great. That's what we preserve, though. I mean, an actual facsimile that's of low resolution and is smaller gets processed faster. And if it's good enough and you're not asking it to do really, really detailed, very fine line work, you're going to want to give it some JPEGs. Obviously, if it's a digitization and it's coming downstream of a digitization work from an archive, we hope that they're doing it to archival standards, obviously, right? But when we have them, we don't want to say, oh, it's in JPEG, it's too low res. You might be able to get the results you want. If I'm in an archive and I have one day in Amsterdam, if I have one day in DC and I'm doing this, right? I mean, and we want to actually do stuff on it, don't say because it's, you know, jpeg that's all but i would say better quality better results but it also

[1:04:46] if it's huge and big and it's 25 megs per that's got to upload it and it's got to be in your server it's got to find it it's got to do stuff to it it's got to read it so that takes time right um sorry and the second question was oh i'm just in general like other other suggestions you might make if somebody taking this on things you learn things you might suggest oh yes and you and you had a question about, oh, so there was like, what do we take on? What are the sort of, how do we experiment? How do we get started? So, so like most of what I started doing was working with Gemma three, which is that Google model I mentioned earlier. It's, it's, you know, uh, 30 gigs, right? So you can run it on a small computer. It's pretty easy to spin up and run it. It's good. It's Google, almost Google good with handwritten text. Um, it's not Gemini. It's definitely not as good as Gemini and, and the newer anthropic models, but it does a really good job of looking at four or five or 10 words like I showed you, and getting some good, reliable text out of it. It's super good at making decisions about extracting structured data from... It's vision-enabled, all that sort of stuff.

[1:05:46] So, like, a vision... Obviously, if you're throwing images at it, you're going to want it to be vision-enabled. The models we used, for most of the really powerful inference, we used Kimi 2.5. That was our big frontier model that's run on eight GPUs at RC. That was super useful. a lot of Qwen models. I use Qwen and Gemma quite a bit. The big bulk of this for a lot of big context windows that we needed, I used Qwen3-VL, full weight. We quantized a couple Qwens down to 8B that were faster doing really small bits of text that needed to be read and summarized in that, summaries and that sort of thing. Making judgments and consensus votes in another project, we use a smaller model as well. Did you shift models as you were moving through? Always. So I I can use this model on with this document. So it's really, really important to say that these models that we use, both in terms of Frontier and, like, if we're using Anthropic and OpenAI and Gemini, we all

[1:06:47] know they do things better and some things weird and some things worse. It's the same thing for the open weights. And so I think some just do not understand what I'm asking it to do, irrespective of it. And one version of one earlier model of it, they just don't get it. And then you have to test them. And that's why, you know, it's lovely to have a smaller model on this LM Studio and just tinker with it until you get something. Then you operationalize it a little bit larger and see what you get. Sometimes you don't get what you want. But it's really cool to be able to just sort of like, I can download this model. It's 250 gigs. Let's see where we go. I can download this one. It's 12. Let's see where we can go. 100%. This is all experimentation. And this is a lot of failure and being frustrated. What's the biggest failure do you think? My biggest failure or the biggest failure where we are now? I mean, of this effort, like one thing you say, man, I wish we could have done that, or it just didn't work, or, again, just sort of running the head of her lessons learned. I think I'll rephrase that because I always want to show like I'm

[1:07:48] being successful here. But I think where I have continually run into really large problems, and I think it's the most exciting frontier, and where we are really getting to the point, I think, very soon that this is going to open up even more is HTR, is the handwritten text recognition. You know, I haven't even had a chance. I know Gemini 4 came out. Super excited to try that out because Gemini 3 .1 Pro was just, like that changed the entire world. That is why I started doing this. I've processed that with HTR handwritten text. And I think that is the next frontier, is how can we start using it for different languages and for handwritten manuscript. We are way past time, y'all. So thanks everyone for being here. Thank you.