Abstract

Three speakers follow one pipeline from page image to query. Jacob Polay compares vision language model optical character recognition (OCR) with Tesseract on early modern and 19th century documents and proposes a tiered workflow that runs on a consumer GPU. Bob Pasker shows why keyword search misses antebellum testimony by Black witnesses and presents Roscoe, a conceptual search index over antebellum court opinions. Keith Pemberton argues that named entity recognition (NER) can only identify a name after it has been read, so it inherits the errors of OCR.

Chapters

Transcript

AI-generated transcript. It was produced from the recording by speech recognition and cleaned up with AI. It has not been edited by the speaker and may contain errors. The captions on YouTube are AI-generated in the same way. Each timestamp links to that moment in the video.

[0:08] can hear me. I saw there's someone in the chat. Perfect. Thank you very much. All right. I suppose we can get started a minute early. So thank you everyone for being here. And thanks to Louis Hyman and all the organizers for putting this together and bringing us here. My name is Jacob Polay. I'm a PhD student at the University of Saskatchewan and what I'm presenting today draws from my master's thesis along with a working paper that I wrote with my supervisor Jim Clifford and other contributors Jessica Jack, Mark Humphries and Lianne Leddy. It's called Reading the Archive by Machine. So I'm the first of three speakers on this panel today. And I'm covering sort of the first step in a digital processing pipeline.

[1:09] So turning an image into an actual page of text, also known as optical character recognition, or OCR. I'm not going to walk you through a specific how to or just give you a leaderboard of all the best OCRs out there. And hopefully from some of the other panels you've been to already, you might kind of expect why. This is much more complex than just giving you the best answer. So I'm gonna try to add a little bit more complexity to it, show you what we found and... So I want to start with this single page here. I'll zoom in so you can see a little bit more what we're dealing with. So you see we have strange spacing.

[2:11] We have images all over the place. We have words going vertically. Basically, this is a broadside from 1700 advertising sugar plums and their medicinal properties. Comes from the early modern corpus that I built for my MA thesis. And it's exactly the document that would trip up any sort of legacy OCR system. I tested it on those and we had plenty of issues. Even the newer models though, I gave it to these models and they all gave me different, weird, complex results. So Chandra 2, which, so I have the CERs here, which is the character error rate. We can see they all did very well, less than 1% error. However, well, no, more than 1%, less than 3% error. But what we can see is things like Chandra 2 modernizing spelling.

[3:16] So this was, throughout the document, sugar is spelled with only one G. This is actually a printing error. The typesetter presumably set two Gs in this one spot, but Chandra removed that. Gemini corrected actually my transcription. I accidentally wrote man or woman as in a woman from a manor, but the actual transcription looking at it back was man or woman. So sometimes it can be better than us. But you know, things like olmOCR 2 transcribed stewed prunes as And InfinityParser, which did quite well in some areas, had a dropped CER score because of it actually kept spacing between letters when I did not in my original transcription.

[4:16] So it put a space between all the letters and directions. And so these are all good, but which one is best? I mean, it would be easy to say Chandra because it had the lowest error rate. But the honest answer is that it depends on what you're actually doing with the text. If you want a keyword search your corpus and compare it to other entities, maybe Chandra's correction of sugar helps you. ...fits which page and what you want to do with it. Since this is a conference about AI, I want to quickly go over what I mean by new model. So every model on the previous slide is a vision language model. the sort of AI behind chatbots just turned to the visual

[5:18] lens. Earlier engines like Tesseract work more line by line, turning the shapes into letters with little sense for what the actual words mean. But vision language models, like other language models, use context. And so we prompt them to do the work, give them instructions, and they read the entire page and use context to help predict some of the errors. So looking at the old problems, the tesseract problems, we had very bad character error rate, word error rate. I tested many other metrics, blue as well, and they were basically all unreliable. percent error rate over my whole corpus from 1612 to 1807. Chandra 2 on the same exact pages only had a 2.6 percent error rate.

[6:24] These same top tools moving to 19th century newspapers got even better and they were within you know 0.6. It's basically a rounding error from our human transcription. So these legacy OCR systems that were used to largely transcribe all of our digitized archives are almost useless. And I'm sure if any of us have gone and tried to keyword search any digital archive, you know that they are quite unreliable. So what we have now is almost clean enough to mine when before it was too broken to even search with. But there are still errors and the errors are weird. So the lines of garbled characters that legacy OCR gave us were obviously wrong. Those errors, while they were numerous, were pretty easy to

[7:27] identify. They were all over the place. What we get now instead are very plausible errors. They're hard to catch. They're real words in plausible places, but they just aren't the right words. In my MA Corpus model, I was transcribing a document with, I caught this as I was comparing these results. The document was about Bristol and it came to the town of Westbury. It saw the W but presumably couldn't figure out exactly what town it is and so rather than telling us that it didn't know what town it was it just guessed and gave me Wooten instead. Looking at 19th century newspapers we see the same thing where it turned Shoal Lake into Sioux Lake in the Saskatchewan Herald, Gotland into Goliath on the same page and doing commodity research I saw rice turned into rye. These are pretty major issues.

[8:30] They don't just give us nothing, they give us the wrong thing. A fluent but wrong transcription is far more dangerous than an obviously broken one. So what can we do about this? There's no single fix but I have four habits that can help us catch most of these errors. So first it seems weird but you can just ask your model for more fidelity. If you just prompt for a diplomatic transcription, it typically gives you better results with no real downside. We did this and and saw a lower error rate in Chandra. Second, just know each tool's habits and this which tool you actually want to use. If you want that diplomatic, if you want the spacing of infinity parser, go with infinity parser and your results will be better if that's what you're looking for.

[9:36] Third, once again, coming to prompting, you need to flag broken outputs automatically and tell it to flag those to you. Otherwise, it will simply give you the wrong transcription and do nothing about it. Another standard thing we see is runaway loops where it just keeps giving you the same words over and over and over again. And so one thing that we developed was if you run Tesseract on the same pages, it's cheap, quick, full of errors, but typically it gives you the right amount of characters. compare the amount of characters on that page with the amount of characters from the VLM ones, you can get a pretty good baseline of if there are any issues. And finally, go back to the page. We need to keep historians in the loop here to check surprises. Here's one final example of this.

[10:40] Jim saw this in the tropical agriculturist. scanning some things and so the scan here from archive.org the actual JPEG it's not good quality but it's okay and the OCR was able to see it and say that it said and we're growing in a bad situation and not robust however the archive.org PDF compressed that to the point where there's basically unreadable text on this page. And instead of saying that to us, Chandra nicely decided to try to figure it out and wrote the exact opposite, said we're now in a full condition and in flower. So not only are all of those previous habits important, but making sure that you have high enough quality scans going in is one of the most important parts here.

[11:41] So getting to leaderboards, how do we actually choose a model? The natural instinct is to just look at what's the best on a leaderboard. But I want to offer two reasons to be careful with this. First is this model GLM OCR. So it's number one on OmniDocBench, which is one of the leading top OCR workbench leaderboards. However, it gets about 10% of characters wrong on any historical we tested on. So these public popular leaderboards aren't very indicative of history. Second is my own MA benchmark. So olmOCR 2 when I first tested this back in December of last year came out on top. Well olmOCR 1 at that point. It came out on top, I built my entire pipeline around the model. Then basically, as soon as I finished my thesis, went back and checked some of

[12:44] these newer models and olmOCR was like fourth place. So, you know, you kind of have to just pick one and go with it at a certain point because there's always going to be this innovation. However, olmOCR 2 completely collapses on newspapers and it's the fastest by So even though it wasn't the best when I retested it, I was still happy with the choice I made because it enabled me to actually get my thesis done on time. So which machine for which page then? That's this is the ultimate question. On clean print pages every tool is basically excellent, so go with the one with the best speed and cost payoffs. olmOCR 2 is a pretty good bet. Early modern print, we would suggest something like Chandra or Gemini if you have the budget.

[13:45] Multicolumn newspapers don't use olmOCR. It's going to fall apart and fail catastrophically. Instead something like Infinity Parser or Chandra if you're looking for something a little bit faster is going to be best. Handwriting Infinity or Gemini again, but you need to review that either way as handwriting is this one area where we do not have great success yet. And finally, unreadable images develop a gold-free failure check that then flags it and sends it back to the historian to check. So our tiered workflow then would suggest running first a pre-run legacy model like Tesseract to get your character count, then run the bulk of your model, bulk of your pages through something fast first like olmOCR. This can run on a gaming PC. I ran this all on an NVIDIA 5070 Ti with about 16 gigabytes of RAM.

[14:47] No need to have a cluster or shell out for something big. Finally then for something with a bit more structure we can go to a more layout aware open model like Chandra 2, and then the really hard ones, we finally reach out to an API, something like a Gemini or a Claude. So what does this change for the historian? First we need to start citing transcriptions. These tools are editing these works just like a translator would, and so we need to write the tool, the version, the prompt, and the date somewhere so that this is recreatable, hopefully. We also need to again, as I say, choose by evidentiary need which tool we're actually using. There's no one good answer. Hopefully with all of this though, and this is something that Jim will be working

[15:50] on soon we can redo the text layer on these giant digital archives and start getting real useful data in them. Fourth, we need to reward honest failure. So the most useful thing is a VLM actually telling us that it doesn't know what it's reading rather than giving us the wrong answer. Then finally, build your own test. make a few gold standard pages, run it on all of these different models and see which works best. That's the best way ultimately for you to know which model to actually use. And if you build that test, feel free to share it with us and we will add it hopefully to some sort of collective benchmark that we're trying to work on so that we all can, again, benchmarks aren't the ultimate goal here, but having a demonstrable

[16:52] benchmark of some sort is very helpful. So with that, I'm going to pass it to the two that actually do something with all this OCR. So thank you very much. I'm Bob Pasker. I'm also a graduate student.

[18:01] I'd like to just take a little poll here. How many of you consider yourselves? You can answer both questions. Historians? A lot. How many of you are archivists? Okay, so we have a, I'm gonna, for the camera here, most of the people are historians, some of them are archivists, so I think we have the right audience here for this. Anyway, I'm a third year graduate student at the CUNY Graduate Center. My focus is on antebellum civil rights, US, and I, in particular, I look at legal history, So one of the things we do for legal history is we look at court cases. And I'm going to take you through how we look at the court cases and how I use them in my research. So back in the old days, some guy would come into the courtroom, he'd sit down, write down what's happening in the court, and

[19:03] scribble it on his notebook. these eventually got printed they took a few years to get into print but they were printed and then distributed then by the time we got to the 20th century we had they've got the box here wrong we had 20,000 volumes and 8 million in opinions and then sometime over the last 10 years, these things got digitized. They were first scanned into pictures and then OCRed, thank you Jacob, and put into a website for people who are into archives. I consider each one of these to be an archival event. That's important here. In the volumes, you would have an index. And I was looking for a testimony

[20:08] by black people in the antebellum period. And so if I go into one of these indexes and I look under the period term for this, I find nothing. But of course, there is a good title for masters and servants, which is not what I wanted but under witnesses there was a one case in this particular volume and you can see that there was one case down here if you have 25,000 volumes that's a quite a bit of searching to do through all of this so we can go we can go up to court listener calm and we can type in a result a keyword search here and you can see the keyword search that I typed in a testimony in black or negro, the period term for this, with the right ors and ands, and for all of you who are experts in Booleans, you can try and get something like this done. But I got 4,700 results on the first time.

[21:09] And among the results were Black and Ward, which was a party to the case named Black, which didn't count. black used as an additive not referring to people and testimony about black people but not by them which is what we really wanted so you can sit there and you can fiddle with these keyword searches all you want and you'll never come up with this phrase at the bottom Negroes when admitted as witnesses it's pretty hard to do a keyword match on that and you'll see that it gets worse Okay, so there was a pretty famous book by Michel-Rolph Trouillot called Silencing the Past, and he talks about how there are silences that are embedded into the historical research process. Moment one is events that happened but were never recorded.

[22:09] Moment two was events that but never made it into the archive. Moment three is when it made it into the archive, but the historian never selected it out of the set of documents to appear in the historical record. And number four is books that were never made it into the so-called canon. And so all of these are moments of silence. And we can see that the historical record, retrospectively, does not necessarily capture I would like to introduce a new moment. How many people have heard of this book by the way? It's pretty popular, good. I'd like to talk to you about another moment of silence which I call moment 2.5 which is works that have been digitized, indexed but can't be found in the archives and they can't be found because some are OCR problems, period

[23:09] terminology and other kinds of issues these things really difficult synonyms we never think of keyword search never found these things all right so the some things to know here just because something has been accessioned into the archives doesn't mean you can find it if you're taking notes all I'm going to put up all the materials they're already up on the web belts there'll be a link for them the second is what I call the Silence of Abundance, 8 million cases, 20 ,000 volumes, impossible to go through. Certainly there are things that are in there that you can't find. And the third is what Susan Grigg back in 1991 said that the finding aids that are used in the archives can only answer the questions that were asked at the time when the finding aid was built. So since 1991, we've come up with new historical questions, and they may not be recorded in the finding aids, and you saw

[24:12] that in the very first index that I showed you earlier. So I built something called Roscoe, and Roscoe is named after Roscoe Pound, who is the first legal historian. And it allows you to type in a conceptual search and find cases. And look at what we found by typing in this query, as legal actors. The first one, Negro evidence, wouldn't come up in a keyword search. This racist term, which I'm not going to repeat, in the second case, the third case has two synonyms for black people. And the fourth one has a synonym for testifying as a legal actor, and that's competent witness. So one query, and you get real results. It doesn't matter if there are spelling areas, doesn't matter if there are OCRs, doesn't matter if there are period terms, it doesn't matter if you have elongated

[25:16] S's that look like F's. All of this is glossed over because the system works based on conceptual search. And the payoff for me was this paper that I wrote last year in which I found 73 opinions people testifying in the antebellum court out of 226,000. That's about one in 3,000 across 11 states and 75 years. And I was able to do it in a couple of days. I call this the hidden archive within the archives, because a lot of what archivists are doing right now is bringing archives of marginalized people into these archives. But you know something? is mainstream arcades. Roscoe is the thing that's going to help you find them. So how do you build conceptual search?

[26:19] Well, first you have to get some documents. The cases that I got, 8 million cases, 226,000 of them were for the antebellum period. I saved myself some time. I only did 226,000 of the cases. They're indexed, sorry, they have licensing terms that I can use, they're freely available, they're OCRed, they're just big, it was 254 gigabytes of text, it's a lot in 8 million cases. Library of Congress, I was able to grab some other collections out of the Library of Congress, I was not able to get out of JSTOR and HathiTrust not because they're not in the public domain Not because they're not OCR because they won't let me they don't want me competing with their search systems So they bar it at the terms of service. So this is something that I think violates academic freedom even though these are

[27:23] Supposedly scholarly academic resources they won't let me run Roscoe on them. And then obviously the last one, no text, no Roscoe right now. So acquire the documents. Prepare them for search. So you have to take these big log documents and you divide them up into passages. And then you take each passage and you convert the passage from text into a mathematical construct called a vector. math is in the next room. And then you take the vectors, you put them in a special kind of database called a vector database, and then they become searchable using some mathematics. We also started with Roscoe also stores all the metadata, the name of the case, the participants, the dates, the jurisdiction, all of that kind of stuff, as well as a permanent link back to the original document.

[28:28] This is kind of more art than science. You can make big passages or little passages, and it depends on what kind of a text you're doing. If it's newspaper articles, usually you can do the whole thing. If it's a book, you need to break them up into chunks. But each chunk hopefully contains some set of concepts, and the search process will find the concept embedded So I designed that, and then I tuned them through validation. I built validation suites that said that things that I expect to find in the database will have rank highly in the scoring, and things that I don't expect to find should rank low. I used basketball terms for 19th century court cases to kind of make sure that the basketball cases didn't come up to the top for whatever reason but validate them and tune the ingestion process.

[29:32] The last part is to build the actual index. It took me 53 hours on that little supercomputer of a laptop sitting over there which has a lot of memory and a lot of AI processors as they're called on it. I had to run it multiple times as you might imagine but it gets stored in the index. The index identifier of the case, look it up in the metadata, the metadata gives you all the metadata and points to the original court case somewhere on the web. It doesn't actually have to store the documents themselves. So how do I know it does a good job? Well, I developed an idea, I'll call it evidentiary fidelity, which includes authenticity of the Provenance of the results and discovery.

[30:34] Does it find things that you want it to find? In terms of authenticity, it hallucinated 0, 163 out of 163. In terms of provenance, all the metadata was recovered correctly. and I was able to point back to the original documents. And then the third one is discovery. This is the first result that I had. 13 out of 20 cases, sorry, 7 out of 20 cases never got found through this set of queries that I developed. That's actually good news for a couple of reasons, but I just want to point out when I tried it by keyword, I found zero, unless I typed in the exact words in the case. Let's see. All right, let me go back to this. The reason why it's seven out of 20 not being found is good news.

[31:36] I was able to track down all seven of the cases that never got found, identify why they weren't found, fixed Roscoe, and then use this as a validation suite going forward to make sure that the fixes that I put in worked, and I can continue to use that as a validation suite and expand it. I did all of this because some of my colleagues were like, you can't use AI to find documents. It's going to hallucinate stuff. It's going to make things up. This is what the AHA is telling us. It doesn't have to be that way. All right, so you can search case law, but there's other kinds of collections that I was interested in. Newspapers, the papers of the founders, lawmaking, and the papers of Supreme Court Justice Taney, who you and so what I built is

[32:37] a what I call Federation. Federation is for all of these individual collections to be housed wherever they need to be and anywhere on the Internet and for Roscoe to be able to query them all simultaneously, collect up all the results and then re-rank them against each other and provide useful information. Roscoe is infinitely scalable like this in the sense that if every library or sorry if every archive had a Roscoe index it would be returned in the results and so you can see the same query here returned interesting results from case law newspapers letters and manuscripts it also means that a little in my little hometown of 10,000 people in New Jersey, the results would come up alongside the results

[33:37] from the Library of Congress. It's democratic, it goes based on the quality of the results, not on the size of the budget or even the accessibility. Okay, what else? So, what does this mean for historians? Search is not a black box. You can't just say, oh, well, here's my LLM, or here's my keyword search engine. I trust it. We're historians. Search is a methodology. We need to interrogate it, and we need to critique it like any other methodology. The second is we need to have criteria for this. I used my evidentiary fidelity for that and actually test it, make sure it works. You heard this from the last discussion. And then the third thing is to prefer inexact results over silence. And we've heard this before, which is you type in a word wrong, you

[34:40] use the wrong spelling or a term that isn't from the period or whatever, and in keyword search, in the results you get zero. So you don't know if the thing you typed in has no results or the query was wrong With conceptual search you get you're always going to get something back. It may be bad Because it doesn't matter like you type in a you know basketball fouls. You're going to get some results back But you'll be able to see what it is. So we prefer inexact results over or complete failure, or collapse as it's called. The third thing, and you heard this from the last speaker, record it, don't just tell us the citations of the case, what tool did you use, and what searches did you make? And the third thing is, sorry, the fifth thing is historians need to own it.

[35:44] This is our methodology, we shouldn't be doing it to turning over responsibility to these four to the engineers. For archivists, the search part is now the finding aid. As we've talked before, the accession can succeed when search fails. And as you heard also before, defects travel. defects will travel. I found luckily that with semantic search we can get past the defects in the sense that mistranslations and wrong spellings, even historical wrong spellings, all of these things are because it's by concept and because it's scored we can transcend most of the defects but they do travel. One of the things I found in my research that I didn't know until I PDF is that

[36:47] there were redactions in them. And so the black bars that were written in some of the cases didn't show up in the OCR text, and I didn't know that there were redactions until I went back and looked at it. So those things travel as well. Licensing. I mean, this is something we really need to talk to the library administrations about, archive administrations. It's in the public domain, and we should be able to use it for research purposes. I really feel really strongly about that. Also images are not text. I did do an experiment which I haven't put into Roscoe, which is using Roscoe on images. I was looking for cases on canal disputes and I was like, well, I wonder if I can find canals on maps.

[37:49] So I took maps and I ran this embedding process on some maps. and I was able to find maps with canals on them in the images. So I can use it for image identification. I think this is great that you'll be able to search for canal disputes. Not only will it come up and say, you know, Erie Canal, but there'll be all the historical maps that show you Erie Canal and you can look at what the dispute is and add that into research. I think I can do that with textiles and other artifacts. But that's to come. Also, some of the benefits of using Roscoe in particular, Federation allows every archive to run there, to keep their own index, build and keep their own index, and Roscoe will interrogate it as part of these searches.

[38:53] And number six is really interesting. the worst, sorry, the poorest cataloged collections get the most benefit. So if you have collections that have been scanned but you don't have good finding aids for or you haven't been able to build a keyword search system for whatever, those get the most benefit from something like Roscoe because it doesn't require, you don't have to go in and tag things and assign LC subject headings and title author all of this stuff is extracted out all becomes available to the conceptual search engine and you're off to the races this is how I was able to find 73 documents in 226,000 opinions anyway okay so that's my TED talk please scan this all the materials are up there including links to the papers that I've written. And if you have questions, we'll hear some later.

[39:54] Thank you. Thank you.

[41:13] Great. Okay. Good afternoon everyone in person and everyone on Zoom. I'm going to be presenting last one of the day for this session, Reading Before Naming, NER, Debt to OCR, and Tiered Pipeline for the Historical Record. And a little bit on myself, so majoring in history at Yale undergraduate, but a lot of coursework in computer science and linguistics, and my focus has been minimal computing toolkits, especially for community archives. We talked about Trouillot, but also thinking about contemporary Michelle Caswell, especially question that really been focused on is how community keep authority and sovereignty over their records as large language models reach for them. Right now I'm on to a gap year dedicated to building this work before going into my senior year during the 2027-2028 academic year at Yale and currently serving on the Society of American Archivists AI Task Force and presented some of this work at AI4LAM and the International Council on archives as well.

[42:15] So the argument really making within this presentation is a name can only be recognized after it has been read, thinking about every person, place, and organization in a machine, built index inherits the errors, the confidence, and the silences of the transcription beneath it. And to kind of set this kind of timeline about the record is older than computing in and of itself, authority control, and the shared authority file, thinking about getting into linking national files, then into 1995, computational named entity recognition, and then finally into the early 21st century, entity linking, thinking about the names and wicked data, geo names and archival authorities like SNAC. Authority files records who institutions choose to describe, which in and of itself, it's this idea of prioritizing either the schemas of the names, the communities, and the peoples who were prioritized by the institution and by themselves, everyone else entered the machine age as like an unlinked string in

[43:19] this transcription. And then like the two histories of, we think about OCR, NER, so thinking about, you know, the optical reading machine in 1929 into 1976, the reading of ordinary typefaces, 95, this kind of first computational NER, and then into the early 2000s, as I was saying, as open source and the transformers raising accuracy on modern text. And then as we go into this last five or six years, newspaper OCR becomes this kind of test set of sequencing of data over time, then evaluations being able to measure damage when it comes to named entity recognition. And then over the past two years, VLMs, you know, reading the page and tagging entities in the same past, which wasn't able to be done before. But the thing is when reading errors become naming errors is I think where this kind of confabulation really starts. So what the OCR reads, the Tiger Seasonal Organization,

[44:21] Fredericksburg, Fredericksburg, the OCR reads incorrectly, no place or a new place which is then incorrect and then also when it comes to names and entities, a broken name or a person who actually never existed in the first place. So the index records what it found and keeps of what was missed. And I think importantly, the categories were built for kind of this newswire. So we're thinking about newspapers, person, location, organization, and miscellaneous. And that miscellaneous category could be so many different entities and kind of subjects that are kind of missing from the record. And people named by first name, for example, ships, plantations, and parishes, acting as places and institutions, one person across spelling, scripts, and decades, in certain kind of years, days, and within calendar. So different variations that the historical records holds, but how that represented in the categories that were initially set don't always make sense.

[45:22] And then when it comes to three ways to joining that kind of reading and naming between OCR and NER, the sequential, so keep the inspectable record, or transcript, and that risk errors passing downstream unseen. you know, the corrected version, so OCR, then model correction, or using improved model over time, then NER, so keeping cleaner text for the tagger, but it risks corrections that invent plausible names. Then the last kind of joint idea of OCR and NER is keeps layout in visual context, but risks the intermediate reading disappearing. And the archival question for all three is, does the pipeline keep the evidence of its own reading at the same time of which in the past And the main thing that remain unsolved in 2025 and 2026, again, fluent misreadings by vision models, returning confident, plausible text on damaged pages, pulled toward the familiar. So based on the trained data and the weights and even the tokens, as we think about going into these large language models, what the model think is correct is really

[46:26] what's corrected into, you know, common things that have been interested into the past. modern benchmarks, so common benchmarks have a recent printed English text, thinking about let's say old English or even scripts that may have less training data from the past, then a long tail of scripts coming from the global majority, so I've been spending a lot of time within my coursework within Arabic, so different scripts of Jawi, Ajami, Pegon, and historical hands having little training data on the web, a lot of that sitting in the archives still currently, And then last problem not so end to end outputs drop coordinates confidence and alternatives over time and Last I think one of the biggest problems is that the clothes dependents So when api retire cost scales with collections restricted records cannot leave So it all occurs within a bubble of sorts of how we think about the transcription into the entity recognition process And kind of what I've been focused on and more so the topic of this presentation is how we think about each page reading it needs.

[47:28] So, you know, measuring script, hand, layout, and image quality, reading that tiered OCR, HTR recognition, naming the entity recognition on a historical schema, which for every document may not be the same, then linking match names to authoritative knowledge, verify checking each claim against the page image, and then query, you know, answer with sources or documented absence, and the idea of router and model registry, learning loops, and provenance on on every output, which the following slides will address. So the idea right now is on Hugging Face, for a lot of those who use it, thousands of open models, it's kind of hard to test everyone, especially for your specific use cases. So the idea of a decision tree, so a pipeline that treats open model ecosystem on Hugging Face as a candidate pool, choose a model per script, per document type, but also a lot of different other classifiers, measured accuracy already, such as character error rate, cost, hardware, speed memory, and the licensing in terms for use and fine-tuning models already, and the selection cycle

[48:31] of searching the hub, test on gold pages, register what wins, retest when new models appear, or the old model you use may no longer be able to be used into the future. In the routing decision tree, a couple of questions that I've been focused on asking, Is there a tested local model for this script? Hand and layout. If no, open vision model, read this page. Is confidence high on the lines that carry names? If no, we read those lines one tier up. Do the authority match the rules and the local tagger agree? If no, send this band up a naming tier. Is the higher tier resolved with confidence? If no, frontier model, then expert review. But if all the answers are yes, except the recorded tier, the confidence, and the alternatives. And the alternatives are referring to running multi-engine. So let's say you run a lot of the different open source models at the same time, preserving the disagreement, but also seeing where the disagreement could be saved going into the future. And this was kind of what I've been running

[49:33] locally when it comes to Tesseract, Kraken, Classic Engines, Fine-Tuned, Recognizers, Open Vision models, but then also the API for frontier models, a page no local models can read reliably and when thinking about reliably that can depend on the script that you're using or the script that you're trying to observe. 80% may be good but also think about the character error rate, the word error rate at the same time. But regards to the tier, expert review, transcriber, specialist, the final reading of contested lines I think is crucial. And the naming tiers when it comes authority and rules so gazetteurs, authority lists, patterns for dates, sums and roles, then fine tune and taggers so small models trained on historical schema for example thinking about ledgers or something that I've been looking into is even like plantation diaries and how those schemas are different than let's say newspapers that have been primarily used for testing in the past. Then open language models run locally prompted with the schema

[50:34] and context so thinking about roles, kinship, unfamiliar categories, and the frontier model is again the last kind of tier, API calls, logged and rare, disagreements, and the hardest spans. And when the tiers agree, confidence rises. When they disagree, the span kind of escalates over time. And just focus more so on the authorities and rules linking names to authoritarian knowledge. So thinking about everything over the past 30, 40 years when it comes to how these sources places have been gathered over time via Library of Congress and thinking about Dorothy Porter but also Getty, GeoNames, Wikidata, SNAC, institutional and community authorities, local finding aids and community naming. But when no authority kind of exists the mention becomes a new entity with its sources attached. So absence from an authority file is evidence too of who institutions choose to describe but also probably what could be what could have been transcribed at the same time.

[51:37] And then the last two things really focused on coming from my computer science coursework is this kind of you know reinforcement training loop of how we think about corrections becoming training data for local models. So thinking about the hard cases that we talked about escalating a strong model or expert answer a person confirms against the page image, confirmed cases, fine-tune the local models. The result is fewer escalations, next pass, and that means let's say less cost towards, let's say, the frontier models at the same time being able to run cheaper and more locally. And then the routing loop, so the router learns where to spend and this is kind of like this kind of harness. So each routing choice is logged over time for institution, its outcomes and cost are scored, the policy favors the cheapest tier that succeeds for each kind of page, and the result is the lower cost per verified page. And the idea is nothing enters training without human confirmation. So keeping communities engaged with their sovereignty over their documents, but

[52:37] also the archives and institutions themselves. And the three kind of principles is local first. So thinking about running on a laptop or one consumer GPU, communities name. So communities saying the categories for their records, authorized names, and then the sovereignty records can remain on-site, models tune on a collection, stay with it, and nothing trains without consent. And back to my original work this year, I've been focused on design first for community archives across the majority where internet access may be unstable or unreliable. So when it comes to offline, kind of accessing semantic search, but also the archives as well. and then mentions first entities later so again like thinking about a deed book a ledger newspaper how one candidate person a cluster drawn over the mentions of one person over time but each mention stays anchored to the characters on the

[53:37] page image uh the archive of the document and just lastly anyone like a you know being at an AI and history conference uh kind of historiographical kind of intersection. Historians have already named the shadows of search so Professor Putnam and the transnational and the text searchable digitized sources and the shadows they cast so keyword search lets historians reach across borders and collections with little friction. The tools hide what would digitize what was read as our last speaker was talking about and what the search missed. Our citations seldom record how digitized source was found. And what it asks of a pipeline like this is records, when pages were read, by which tier, and what confidence, and makes every result citable back to its page and reading and report what was searched and what was never read. And lastly, each step is an archival event. So image to text, text to mention, mention the entity, entity to query, this idea of this kind of archival process staying tied to each transformation but again what

[54:42] can be lost and what has been lost is the idea of layout for example with newspapers the idea of sequencing so if let's say a news article goes from one column to the next how that could be lost within previous OCR runs but also context disagreement between readings everything that was never read a document recorded absence going back to Trouillot's Silencing the Past and how important that is. And when a search returns nothing, the historian should know what nothing it is. Absent from the record, present and read with low confidence, or present and never read at all. And I think those are the three ending points that are crucial as we think about this work going forward. Thank you. I think we technically have about five minutes for questions. So yeah, I think we have time for like probably two.

[55:44] Great, I'll keep it quick. Thanks so much for the super panel. This is for Keith. Yeah. First of all, great presentation. Thank you. Very impressive. But I'm wondering about, I'm just kind of trying to pick your brain as a science person, Yeah. whether you think there's any possibility of improving the performance of neural models like Tesseract, can we use LLMs to improve the performance of those models? That would be very economical. If you could actually just get the output quality of those cheap, expensive, small models to perform better, Do you think that's possible from a computer science perspective, or are we just at the apex of performance for those models? I would say we haven't even, I feel like, reached the halfway point of what those models can do. I think a lot of, when it comes to, let's say, the transcription or even the readings of things that aren't being read, let's say, well by Gemini 3.5

[56:46] Flash or even, I think Opus 5.5 just came out as well. So it's a matter of how we think about this synthetic data generation of languages that don't have a lot of web content. And I know there was a whole incident with the Internet Archive and how their data was being kind of leaked towards OpenAI or towards Anthropic. But there's a way to do that, I think, in a more ethical way for, let's say, scripts or even multimodal data. I think that's where these models can improve. but I know there's some startups working like a neural symbolic AI that may have a better neural net I guess going forward so I want to say like this is the only option but there's different things popping up across the scene right now Yeah, yeah, just a quick question. I think a couple of you use this term gold pages, which I haven't come across. And can you define that?

[57:48] Yeah, do you want me to take it? Sure. I can. Oh, yeah, yeah, absolutely. We'll just share. Hopefully the people online can hear me. So the question was can I define gold standard? Yeah, so that from my use of it, it was just pages that I personally transcribed by hand creating essentially a single key verified standard that I could then compare these models against. So If I had the resources and I wasn't trying to finish a master's thesis, I would have double-keyed them, gotten somebody else in the department or somebody to verify and combine. But with the resources I had, I hand-transcribed them, and then that was kind of my set with which I tested everything against.

[59:14] thank you in that space, how do I respect the structures of kinship when a lot of these data search terms are not respected? So just a little bit of an observation, so I think that's a good powerful part of the presentation. And it would be great to hear more about that in the future. Absolutely. Thank you. All right. Well, I think we're at our time here. But if anybody has any other questions for any of us, we will, I think, be around. Yeah. I know it's a quick question, so feel free to delegate. But I've been interested in this question. I asked this to We had a fantastic presentation on a similar subject. What's the possibility of these systems, and I know very little about the engineering part of it, to translate the terms that you

[1:00:15] find with such great efficacy and precision into actual types of knowledge, like tacit knowledge, as opposed to just explicit knowledge? Is there a possibility of doing this at all? Have you thought about it? Or this may be obvious to those of the core experts, but as a historian who's dealing with other things, I don't know. Keith, do you want to? Yes, yes. I would say like... Here, do you want the mic? Yeah. I would say the... Probably like an objective for something like this would be to like have someone who has no familiarity with, let's say, any type of software engineering background or even like, I guess, computing background, even doing genealogy or even, let's say, halfway across the world to be able to like interact with something like this and then like kind of... their own schemas as well. Is that, I guess, what your question? In ontology. On ontology, yes, yes, yes. Okay. That's what I meant.

[1:01:15] I don't know if this is a layer that you guys deal with, if that's something that people plug in already somewhere, or maybe you. So with the New Haven, like Port Authority, I've been working on like the maritime records in New Haven, so they've been helping create a schema for that to apply it. But I think at scale would be something that I think would be transformational for any, but also anthropologists, but also communities, and plugging in within their own scripts, within their own, let's say, even audiovisual material, I think could be on the horizon.