Abstract

A Brown University team asks whether AI (artificial intelligence) handwriting recognition is solved, running handwriting tools such as Transkribus and then frontier and open-weight models on three collections: excavation notebooks from Abydos in Egypt, Wilson Harris's manuscript notebooks, and the 17th-century letters of Maria Madelena von Habsburg in Florence. The models struggled with locus numbers and with Harris's overwritten drafts, and on 87 hand-verified pages of Habsburg letters Claude Fable did best while Opus 5 fell from first to last under diplomatic transcription rules because it hallucinated. Price explains why a low word error rate can still hide errors that change meaning, such as gender agreement in Italian. The Q&A covers attributing marginalia to John Major and the ethics of giving apprenticeship work to a chatbot.

Chapters

Transcript

AI-generated transcript. It was produced from the recording by speech recognition and cleaned up with AI. It has not been edited by the speaker and may contain errors. The captions on YouTube are AI-generated in the same way. Each timestamp links to that moment in the video.

[0:06] And mine is on mute. We'll do better. Okay, we have a microphone now. Yes, yes. Okay, so we have Ashley Champagne, we have Anya Lewis-Meeks. Hi, I'm Haley Price. I'm a fifth-year PhD student in the History Department at Brown University. and we're going to be talking to you all today about our experiences in a group project on AI transcription and what our findings are and yeah we've got we've got some kind of mixed conclusions so I think it'll be interesting and we'd love to continue the conversation after the talk with you guys. So yeah I'll just do an introduction so our Center for Digital Scholarship has run an AI and experiments in digital scholarship project series since 2022. So we pick a new project essentially every year. And this project came about this past year.

[1:06] So Dan Cohen, the vice provost for information collaboration, dean of the libraries and professor of history at Northeastern, wrote that one of the hardest problems in digital humanities, handwriting recognition had been solved. So at the time, our faculty director, because CDS always has a faculty director that I work with, Tara Nummedal, who's an associate professor of history at Brown, thought, you know, really has it. We have so many, you know, different types of handwriting. There's so many different challenges that AI would face. And so we wanted to test this hypothesis. So we wrote a call for projects at Brown to see if we could get together a team. And we really got together the dream team. So we, here with me is Haley Price and Anya, but we also have a PI, Laurel Bestock, who's in Egyptology, a professor

[2:09] of Egyptology. So interestingly, also, like our PIs are a faculty director, a graduate student, and a postdoc. So lots of different levels and also different disciplines. And that was important for the project also because we really wanted a diversity in terms of the materials we were working with. And then also on our CDS team, we have several people. So Cody Carvel is our technologist, and they have done a lot of the work on this project to work with different AI models. Tareka Sankar is the project manager. Elizabeth Yalkit also does a lot of the technologist work. And there's me. So our big question here is how do different AI tools, we wanted to try a lot of them, how do they perform on hand-raiding recognition across a widely diverse rapport? And so we used a variety of AI tools. We started first with handwriting recognition tools because we thought that's

[3:11] exactly what we're doing. And so that will make sense. Also, there were ethical reasons for doing so. So Transkribus is nonprofit. It puts cultural heritage institutions, libraries, archives, researchers in direct control of the technology. Leo is co-founded by a historian. We had a really interesting call with him. And so we thought that that might work. And Archives Studio is in a prototype phase with Mark Humphries. And so we hope to use that in the future. but quickly kind of moved more towards frontier APIs and trying those out. So as well as open weight models that publish a lot of times there, how they weight the outputs and their code sometimes. So we thought we needed to determine how we were going to evaluate the tools. We wanted to try this range of tools, but we also wanted to ask of these list of

[4:12] tools, can the tool be trained to stop when unsure about a letter rather than just continuing or hallucinating or being so confident about something that it shouldn't be confident about? Can we trust it to transcribe confidently? What ethical framework does it use? So I mentioned like Transkribus and Leo seem to have like ethical frameworks, but we also wondered, are the models trained on works under copyright? Are they usable? Is there a transplant parent policy for how the tool uses user data? And we have a number of other questions here. We also wondered, can it be trained to improve? And then we also wanted to just not evaluate the tool, but we wanted to evaluate what it would output, the results of what it said. So we wanted to ask, you know, is it useful for the project? Are the transcriptions, are they like the human transcription that we would

[5:13] ordinarily get? Is it saving us time? Is the research workflow improved? Does it expand our own human thinking? And I think this is, at least for me, a very central question in this project is like when you're using AI, does it expand your human thinking? Does it make the scholarly process richer? That introspective question. And so here I just want to give an overview of three of the projects. So the first, which is Laurel Bestock's project, she came to us and asked, like, I have 30 years of archival notebooks, and could AI help speed up the process. This is a challenge because there's a number of field-specific annotation conventions. You can see here there's some scribbles, there's triangles, or called a locus. There's different types of handwriting.

[6:17] We asked, yeah, what if the handwriting is unintelligible? In one recent NEH-supported project, students were able to decipher a 2,000-year -old scrolls for the first time. So we're wondering, you know, what are the possibilities? What if the corpus is multilingual from a variety of geographic areas or time periods? How does it perform across those? So Cody Carvel, our technologist, wanted to try out the different models by producing a toolkit. Essentially, I can't speak to the the very specifics of the technical details of it. But basically what it is trying to do is try out all kinds of these different AI models and compare the output together to see what, if the AI is seeing, if the different AIs are seeing the same thing, if they're finding the same errors, which we did find in one of our projects, or what the differences in the output are.

[7:27] So Laurel Bestock has, I mentioned, this amazing archaeological archive that's held in New York at the Institute of Fine Arts, which is part of NYU. And it includes archaeological field records from 30 years of excavation conducted at the site of Abydos in Egypt. So these records include excavator notebooks, hand-drawn plans, artifact drawings, slide and digital photographs. The vast majority of these records were not published. So when she took on this role to be the director, she decided she wanted to slow down the process and make these archaeological records more easy for researchers to use. So she wanted to do a lot of publishing of what was in the materials. And so she asked us if we could work with her on the transcriptions to

[8:27] make it faster. She's currently in the 1990s, so working her way up. So you can imagine the work that's required of this. So yeah, that was her central question is just, can it speed it up? So this is a sense of the field notebook. So each excavator notebook contains a daily narrative describing what happened in the trench. So this is called an OP. It includes plans of progress, and it logs assigned numbers, which are also used in the narratives. So the number types include a loci or archaeological context, lots, which are subdivisions of the loci, artifact numbers, elevations, either in lists or marked on plans or photography numbers.

[9:28] and all of these numbers are how researchers will then use the materials that are found so Laurel Bestock provided this example that when when you might be digging a 10 by 10 you might subdivide that into different lots and you might find that the material culture that you find is a little different even if the sand if it's sand all looks the same and that to an archaeologist is like, well, then maybe the sand was laid at different times. And so that's why all of these numbers are very, very important. And so there's a ton of conventions. And then even further, you have the delta, which is the triangle. the delta is used to mark the locus numbers um which is the site circles are used for elevation dates and time numbers are written in roman numerals um

[10:30] and that's also interesting because americans write month and day and then europeans write day and month so in these in these notebooks, you have different conventions. And so what they've done is then write in Roman numerals for the month so that at least then you could always distinguish what is the month versus the day. So it's all very different, but it doesn't always follow exactly that pattern. So there's all kinds of these challenges when you're trying to then take this handwriting and transcribe it. They're also using Arabic a lot. Laurel Bestock said that, you know, a lot of times there's not a term in English for the type of soil they're seeing, and so they use Arabic. So then each of the notebooks need to be put in structured data into their database, which they're using a tool called Kiosk, which we have at Brown, and they move the transcribed text into the database.

[11:33] And then they need UUIDs because every project has a loci 15. And so to be able to figure out what is loci 15 in this context, you have to assign a UID. And so all along the way, Laurel Bestock was asking, you know, humans make big errors too when we incorporate AI into this process, what new errors might enter the process, what errors might it help us fix. And so essentially with all of these, we tried to find a notebook that was the most accurate, that was human created, and try to use that as our ground truth. and then developed a rule set to guide the different AI tools to help it hopefully transcribe the materials in the best way. So it had a really, really hard time with these

[12:36] numbers, with triangles, with circles. And so this rule set essentially says a triangle is a triangle. This is what these conventions are. But it really struggled with these. And it was only in, I think Opus 5 was one of the models that was the only model that ever wrote locus question mark instead of guessing confidently. And so, again, here throughout the process, I think Laurel asked a really interesting question, which is like, when human data intros are reduced to number checking, essentially, is their own interest or investment diminished? Because there's a lot of checking of the AI. is there any trade-off there that's worth their time? And so we essentially found, you know, numbers are a big problem. We did troubleshoot this. I was emailing with Cody Carvel earlier today, and I was

[13:39] like, you know, do we try this? We try that. We, you know, Cody tried a lot of different things to work with it, but it's a huge time investment to train the AI. but the locus numbers were really, really challenging. So essentially with Laurel Bestock, we thought, this is Laurel Bestock's thought here, that like it could really help with transcribing blocks of text, but it had a really hard time with these character level mistakes. And those are huge for her and her team when she's trying to analyze this. And I will turn it to Anya. Oh, and my mic.

[14:40] hopefully that works um so my project is a little different i think from the other ones on our panel today um i'm a postdoc in the literary arts department at brown which is our creative writing department and i'm an english phd um and i was just talking with some historians today who referred to what I think of as older work as what a joke. Not quite, not quite. So Wilson Harris, born in 1921 and died in 2018, is a Guyanese writer who writes what he calls quantum fiction. And it's very kind of complex philosophical work. And I, in doing my kind of research into his novels, decided to look at his kind of manuscript drafts to get a sense of what he refers to as his revisionary project or process.

[15:40] So in 2023, I went to the Harry Ransom archives in Austin, Texas, seeking close engagement with his manuscripts and drafts, imagining myself almost operating as a detective, deeply investigating these handwritten drafts for intellectual clues. And I began with one of Harris's quote, holograph notebooks and found myself scarcely able to decipher his handwriting. As you can see, there's kind of what looks like frantic scribblings and scratchings. The draft has been written and overwritten. Text has been crossed out and effaced such that it was hardly legible. And so I approached Ashley at the CDS at some of an impasse, having presented my research at the 2026 American Comparative Literature Association conference. I was interested in tracking the progression of drafts for just one of Harris's stories, Yuracon, from a manuscript published in 1969

[16:44] called The Sleepers of Roraima. But even in that case, things proved to be challenging as drafts that were originally called Yuracon became Aikuyumucan, became Kuvad, which were all the kind of three stories in the book. And part of his revisionary process would be switching out characters, content, stories kind of across the draft. So there's a lot of change happening here. So my goal for the project changed and I, instead of overall trying to see what happens from draft to draft, simply started to ask the question like, can we figure out what this man is writing? And can these notebooks be turned into something legible at all? So Cody Carvel, our technologist, as Ashley referred to before, ran a number of different models, including Gemini 3.8, Flash, GPT

[17:44] -6, Astra, Muse Spark, two versions of Opus against these or on these notebook drafts, as well as on the kind of manuscript drafts that exist in the archive. uh they ran two transcriptions um one that they called diplomatic and one that was aggressive um giving the models the opportunity to guess where they could not make accurate transcriptions and i attempted to provide a ground truth on a much more uh legible document this is a letter from Harris to his close friend and editor. And interestingly enough, so I'm not a paleographer, and presumably at least Harris was able to analyze his own handwriting. If you Google Wilson Harris handwriting difficulties, it will tell you that his wife transcribed, or Gemini will tell you that his wife transcribed his materials,

[18:49] but I couldn't find any evidence for that. Just a little, some of our issues that might come up later in our discussion. And then Cody interpreted through the different models transcriptions through Claude, and these were some of their findings. So even for the ground truth, there were major amongst the models. This is for the notebooks, but for the ground truth, for instance, there were several solo misreadings, so on proper nouns and rare words. Something that said Curacao became Greece. Austin became Andean. Augur was engaged. So even at the level where I had managed to do that initial transcription, the model still found errors. And they did manage to note some things. Like for instance, I had neglected to transcribe the little letterhead

[19:53] and they all told me that I didn't do that. So they caught a human error there, but there are still issues. And for the notebooks where there were no ground truths, again, because I couldn't read them, um the kind of main concern was that there were just massive discrepancies across each of the novels or models abilities to interpret the material um and such that the conclusion was that um no human no model can be trusted on what there was without a human reader so again i had a ground truth for the letters all the models agreed that that ground truth could not be used for the notebooks, even though presumably it's the same writer, same handwriting, they weren't able to do that. We also found that the runs weren't repeatable. So you couldn't run a model multiple times on the same notebooks and get the same answers. Deletions ran inconsistently, and pretty much half of every single page was

[20:55] in dispute. And this was Claude's framework. sit on the non-letter Harris material without a human reader as the solution. So, oh, and then of course, each of the different models kind of had different issues with them. We found that Gemini 3.8 Flash had the most readable, aggressive text, but it also would be more willing to make guesses without knowing whether or not it was correct. Claude called it fluent, sometimes confidently wrong. And so this question of kind of legibility still remained by the end of the project. So I have a secondary question that emerges again, and walk with me here from being a writer and somebody interested in this kind of philosophical, almost spiritual

[21:57] figure and decided to pose what Wilson Harris includes in the opening of his archive to the book Carnival, where he talks about this revisionary process kind of in its sense or as a whole. And so Harris's reading is, my intuitive sensation is that though I am in cold Logic, the author of the book, Other Nameless Hands, wrote through or in me imbued with different dimensionalities, different time scales to overturn linear complacency and create self-reversible mutualities between author and living text. So across drafts, what happens for Harris is some kind of almost like spiritual access of different hands. So I posed the question to the models, were they able to kind of identify any understanding of this writing process in

[23:00] the revision process, or in the revisions in the notebook drafts? So my questions were, is there evidence in the style of handwriting that this is a single author text? What elements of the text may suggest the presence of one or more authors, which became agents? Are there multiple timescales that work in the text? Multiple dimensionality is present. Can you identify more than one agent? And so we did this on these monograph drafts, which as you might notice are maybe even more hard to read than than those for the stories, which proved, again, so just another little annoyance. This is the process in the composition of Carnival. Presumably, it is a process that you might be able to use to think about his work in generally. The models wouldn't allow for that. They said it can only be done for this one text. So that's kind of a logic reason question.

[24:03] and the results were mixed as you might imagine via Claude. What the images are and what they are not. This essay is about instruments and evidence. Three vision models examine the same 21 photographs under one rubric. Each was asked to treat Wilson Harris's 1991 note on visionary cycles in the composition of Carnival as a hypothesis to test, not a fact to confirm, etc. And so the results were inconclusive. On the question of nameless hands, no one model was able to return an answer with any confidence. On the question of different timescales, however, Gemini 3.8 Flash supports evidence for timescales with high confidence. Dimensionalities was given insufficient evidence by a number of models, but Gemini Flash noted many realist to visionary shifts. And just in general, Gemini Flash was the most willing to support and to contradict

[25:08] and reads confidently where others cannot. And Claude's kind of interpretation was to weight it down accordingly, even though one out of three models seems to me kind of a certain, a significant number. So I think it's interesting to pose kind of these questions of these models almost as scholarly interlocutors, asking them to move beyond this initial kind of question of transcription into this interpretive mode and see what the limits are and where we ran up against them. And maybe also thinking about when we're asking students this work, where are they asking the model to kind of go beyond their capabilities? So I close here with this concept of opacity, which is an important theoretical concept in Caribbean literature that basically seeks to kind of move

[26:08] away from making things legible and understandable and allow for that moment of, yeah, lacking in certainty and I don't know. I ended this project essentially feeling that I hadn't gotten any closer to understanding this writer and maybe that was okay. Um, but as a, as a, uh, or maybe I understood opacity more, we'll put, we'll put it that way. Um, so in terms of accuracy, inconsistent across all three models, when it comes to this kind of question of philosophy, uh, there's still opaqueness. But again, as a writer, scholar, maybe accuracy is not as important as thinking about process and imagining kind of this conversation to be had with potential scholarly interlocutors, if we can, might think about these models in this way. Okay. And there's sort of a little bit more about the rule set, how

[27:14] Cody attempted to work with deletions on multiple rounds of drafts, so on and so forth, and what some of the challenges were. But I'll turn it over to Haley. Thank you. Hi, everybody. Again, I'm Haley Price. I'm a history PhD student at Brown. I'll give you just a little bit of context about sort of who I am as a scholar, because I think one of the really interesting things about this project is that all of us are so different. We're at different stages of our career. We're in different departments. We have vastly different projects and different sources, and I think that's really cool. I also would like to point out that in some ways our presentation is a little bit of a disappointment because we all kind of found

[28:14] that AI didn't do what we hoped it would do. We didn't have the results that we necessarily wanted with our transcriptions. I'll tell you exactly what I'm disappointed with. But I would like to say, so I'm a digital scholar. That's sort of my main methodological identity at home. And I've found great success with AI in lots of AI or a stance of CDS or Brown on AI. This is just one experiment where we were really excited to see, okay, we would love transcription to be solved. Is that true? And unfortunately, not yet, but it can be. So where I came into this project specifically is it's a subsection of my dissertation. So my dissertation is I'm an early modern Italian historian. I'm working on this Maria Madelena von Habsburg, and she married into the Medici family. So she is, by blood, one of the most prominent families in Europe during the early

[29:17] 1600s. And then she marries into one of the most prominent Italian families. So you can imagine that she knows everybody who's anybody. She's related to so many important people at every major court across Europe. So her network's fascinating, and she has so many letters. And compared to the Medici men, Pretty understudied. Not like an obscure figure, not totally understudied, but understudied. So my dissertation looks at her letters. I'm doing a network analysis for her. I'm doing a lot of data viz stuff. For the purposes of this AI experiment, I took a sample of 100 different manuscript letters from the Archivio di Stato di Firenze, where all of her letters are kept, which is kind of a hybrid family archive of Medici and state archive of the Grand Decide of Florence. So it's an interesting space. I basically wanted to throw spaghetti at the wall and see what's stuck. My sources are super diverse. This is a German woman with family all over Europe who's living and ruling in Italy.

[30:18] So I've got a ton of different kinds of sources. And I'm just going to show you a sample of kind of what we're working with. Who in the room works with pre-modern manuscripts? Yeah, we have two. Two. Incredible. Maybe a third. I'm also going to be old enough that I'm not confident enough in the definition of that word to agree or disagree. Incredible. I'm going to... I'm going to... I'm going to... I'm going to... I'm going to... I'm going to... I'm going to... I don't know. I'll count you as one of us. I don't know. Well, we've got maybe three, maybe two. But this, not normal. Gorgeous. Easy to read compared to a lot of the handwriting that I'm working with. But if you don't speak these languages, complete mess. You know? And then, so you can imagine, you know, ChatGPT will have to be good or Gemini or any AI model will have to be good at a lot of different kinds of handwritings and languages to be able to dissect those. Same with challenges like this.

[31:22] We've got holes, you know, documents. I mean, these are 400 years old. Like, things happen. Preservation isn't perfect. Pages get ripped. They get water stained. Archives experience fires and natural disasters. I mean, these are pieces of paper. They're sturdy paper, but they're paper. So how does AI deal with that? How does AI deal with different handwriting across letters and even within one letter? It's a challenge. It's a challenge for a human. It's a challenge for AI. And I've played a lot of that. now these are some of the less clear examples they're not maybe the worst examples of my whole data set and my whole dissertation project but significantly less readable this has bleed through coming from the other side that's an additional challenge this has bleed through and it's in german which i don't know what the germans were doing but they have decided to make handwriting as difficult as possible in the 17th century. You're disagreeing. I assume you can read it well. But this, for me, is a real challenge because I don't speak German and I'm not

[32:24] familiar with this particular kind of paleography. I do work a little bit with ciphers, which show up a lot in Italian correspondence at this time, particularly in marriage negotiations is where I've been seeing them. But we go back to Laurel Bestock's project, Problems with Numbers, switching between identifying letters and identifying numbers for the AI, an additional challenge. So none of these are insurmountable challenges for a human or for a computer, but it's a lot of challenges to throw at them. As I said, I've got five different languages that I'm dealing with. I read two of them. I read Italian, I read Spanish. French is similar enough that I kind of get it. I can kind of make it out. German's really hard for me, And as any of you who know Latin will know, different kinds of Latin, different time periods, you know, even if I had a general Latin or a classical Latin education, I don't, it's hard.

[33:25] So I'm going to invite Ashley to come up and help me explain some of the work that Cody did with my work and some of the challenges that we face. And I'll kind of chime in and help fill in some of the blanks. yeah um yeah so once again you know provided a rule set um to offer ai some context here um some of the known challenges um our abbreviations are common i'm saying this like as haley as this is you know your research but it's how your abbreviations are common um and uh so it notes those the other challenges it notes that there's ink fading and bleed through that are common in documents? Oh, thank you. It was not on, sorry. Thank you. Really bad with that. Just repeated brief. So, so a couple of challenges are there's there's

[34:25] abbreviations that are common in Italian and Latin and, and also there's a lot of ink fading and bleed through in these texts. And so the project tested AI models. I mean, I thought this was pretty interesting on a hundred pages of difficult 17th century handwriting featuring these challenges with multiple languages and heavy abbreviations. But the team used 87 pages that were verified. Yeah, I hand transcribed as much as I could. And then I got to the Latin and German and said, oh, no. So I didn't do those ones. So that's a pretty strong ground truth that we had for this project in comparison to onions and laurel best stocks. So Claude Fable won by a landslide. These are Cody's notes here. And we also found that a high cost doesn't mean necessarily a better transcription. So we tried lots of different models. And that wasn't the most expensive one.

[35:28] Muse Spark performed very well. a quarter of the cost of GPT -6 Astra, which was also tried. A few other just interesting, a little bit random notes. When we switch to diplomatic transcription rules or a literal exact translation rather than adding context as AI often does, opus 5 dropped from first to last because it hallucinated. um this is also interesting because we we noted a single uh every model failed on a certain image which showed the human uh ground truth error rather than ai failure yes which also came up in in anya's piece too yeah so a lot of challenges um some things that uh i was sort of frustrated with as a historian, I, you know, it's, I'm responsible at the end of the day, of course, for all of the interpretations

[36:30] that I make in my dissertation, all my arguments. I want to be able to read the sources and know for sure that they're accurate. And I know exactly what the words are. Like I would never try to make an interpretation and publish something if I hadn't read the primary source myself, that I just wouldn't feel comfortable with it. So when we say AI was, you know, better at some things worse at other things, like what exactly does that mean? We talked a lot, Cody and Elizabeth and the team and I about, sorry, just getting all this stuff up, about word error rates and character error rates. And that's sort of one way that people measure how accurate AI transcriptions are. And that just means word for word, how many words are right, character for character, how many characters are right. And for me, you know, even if it has a high percentage, a really good word error rate. There are very few word level errors. Some of those words are really important and affect the interpretation. And for a language like Italian or romance languages that do a lot with gender and number agreements that are dependent on just

[37:34] an O versus an A, completely changes who I think the letter is addressing. So for me, I'm not comfortable using AI transcriptions yet. I would love to be, but that's not quite where we are. I could see where this could be really, really useful if my massive corpus of thousands of letters could be transcribed quickly. But right now it's really, really time intensive. So we could do the things that Cody did and run scripts and not really hold the AI's hand, or we could do things that I did, which was hold the AI's hand and like really mentor it through one handwriting really well. But you saw how many handwritings I have. And that was just a small sample. So really, really interesting findings, but it really just raised a lot of questions for us, a lot more questions than answers. Not quite, sorry. I'll be really quick. So yeah, so our conclusions, we wanted to know how helpful improve the scholarly

[38:39] process. A very open question. Curious what is happening with your research. Does it make it better? And then my big question also in this was how does this change our work as an AI center? Oh my gosh. As a center for digital scholarship where we're always working with faculty. And faculty now are vibe coding entire digital scholarship projects. We've seen that. We've just produced a vibe coding project. What role does the AI, does the Digital Scholarship Center play then? And so what I think though, and what I think we've come to as a group together is that the Digital Scholarship Center, I mean, at the core of the research, very much like the discipline of history, it's very much about collaboration and about talking through your ideas and putting

[39:39] together something that's valuable for your discipline, for the research, and for the world. And so I think this project shows we were able to come to that and our other projects in the center show that, that that communication and that collaboration is really, really central to these kinds of projects. So I hope that that will move forward. And I don't believe that AI will replace that at this point at all. So thank you. Thank you all. And we'll bring you guys up. Yeah. Yeah, we'll take questions. a PhD candidate at Delta 11 sites. So kind of coming at this from diplomatic history, national relations, diplomatic approach as a more like soft approach rather than a more aggressive one. So I'm at this conference because I cold emailed Lou about a handwritten transcription question. So I have 50,000 photos on my hard drive that I took in the

[40:44] National Archives in Britain and France and presidential libraries in the US. There's a lot more annotation in the British system, a lot more marginalia. So I was basically doing a workflow using a kind of good enough Tesseract transcription, but then layering a ad hoc final layer of a Gemini flash query per page, but there was marginalia or garble or something. And in the course of doing this, I came across a very funny and catty annotation, which was at this point out of power, Margaret Thatcher, writing to her successor, Prime Minister John Major, and she's saying, you're not doing nearly enough to help the Kurdish people of Iraq. on this, or this is the question, was that him? Was that him? Was that the secretary? And so this is a different question from just transcription, oh God, very clear, clear as day. You know, they caught oh God immediately. The question is, is that his handwriting? So I'd be challenged it, is this John Major's hand? What examples do we have where it's incredibly clear from context, it is John Major. What's the ambiguity? What is the style? And it's like, okay, it's a blue pen.

[41:47] That's not just positive, the secretary office is blue pen. He writes the date as month dot day. The secretaries are using month slash day So the dot appears to be only him if it was loop over the O. So after a first pass I was 70% confident now I'm 95% confident Here's an index with 868 examples of this handwrite A from the rest of the corpus And so I'm going through checking this as well as I can, but I'm not a forensic handwrite A panel I think as we're talking about with machine translation and transcription and realistically speaking we're not going to hire a human transitor to translate tens of thousands of patients we can't right we can zoom in on the one example that we pulled out with the machine so i think the idea that it points you toward the needle in the haystack so i'm thinking of doing some kind of methods piece either for some of the journals run by some of the people here right on historical methods from the journal of digital history um titled oh god on this question i'm trying to puzzle out um so even much more in the handwriting even more difficult material this is 1990s, you know, telegraphs and kind of like somewhat digital, somewhat

[42:48] handwritten, but with the marginalia of the question. So it's not cost prohibitive to do ad hoc Gemini and Lash. But yeah, I guess you were asking the question, not just philosophically, but practically, was there another hand at work in this, right? The annotations, all the main author or perhaps someone else. I think like when you have clear examples of one author, so this is just attribution. The second question is, in talking to colleagues about this, they said things like, okay, so I assume what you've done on this is found a manual sort of review of just one document set where you transcribe 100 pages of physical standard and then it uses that to create a custom rule set. I'm saying, no, I've just been letting Opus 5.5 with sub-agents and the Gemini API plug-in. I've not been doing an ad hoc per corpus rule set. elaborate German handwriting of the 17th century. It's a different story. But I am curious about the general versus the specific.

[43:49] I think you've seen a turnover just in the past year and a half of the general models performed a lot better. Good to know that Opus can still hallucinate. I haven't filled my spot checks yet, but I'm still wary of this. Anyway, that's a big data dump. But very excited that you're looking on this to look at correspondence more after. Where did you land? I landed. I'm pretty sure I think no one else would have been so obnoxious to write, oh God, on prime ministerial correspondence. Yeah. So I think it's part of the context that he's fed up. It's like, stop harassing me about human rights for the Kurdish people. Jesus Christ. You know, it seems pretty clear that he's he's annoyed that he's being needled by this point. Retired Thatcher. Yeah. But what came across is my my work is about U.S., U.K. and French intervention in northern Iraq. No flies out against Saddam Hussein in 1991. But he's like, I did the thing. OK, oh, God. Yeah. I mean, your project sounds really fascinating. I want to make sure we have time to get to other people's questions as well. and then one there in the back. Thank you very much. I have two questions. One is you briefly touched on things like transcribers, which I'm not a

[44:53] huge fan of in general. But I think for tasks like yours, it might be actually a better way to go. Because if you are able to provide the ground through actually you can train the model. This is a little bit related. what is your, I mean, listen to it. I had a hard time hearing, working with data where you can't read what it's in. I think that it's an issue. It can be solved by someone else that gets hired and reads it. But like, I think these workflows make just sense if you have some kind of ground truth. If you can read it yourself or someone else, but like, how did you, in the end, or how do you want to go forward dealing with these issues? Yeah, and I mean, oh, great. Right, right, right. So the question was, like, if presumably that there is somebody out there who can read this work, that person just doesn't happen to be me. And so should that person do it?

[45:55] And where do I move not being that person? Was that sort of the question? If you hire an undergrad historian to tell you geographic classes, they can read this. It was not that difficult. I'm not saying. Yeah, yeah, yeah. Yeah, that's, that's possible. I think the question comes down to, um, but could a model like do that instead? Um, and, uh, if you again can train it on that initial ground truth, which is why I guess I found it interesting that still using a version of any handwriting of this writer couldn't do that. Um, I, I guess that's where, that's where I landed in it at least making sense to sort of compare the two things together. But even like when I first came to this project, and it's funny because I've presented, you know, sort of versions from this several times. And the question has been like, oh, like you could get there. It would take time. And I think maybe that's sort of what all of our projects have

[47:00] in common. Like how much should this be saving us And is it or not? I don't know that it wouldn't have a similar effect, maybe if one notebook had been done by an undergrad, as you pointed out, who's taken paleographic classes. But I didn't have that luxury in this semester. And so it wasn't helpful. I don't. Yeah, that makes sense. Absolutely. Just to say, Laurel couldn't be here today, but she does have a small army of undergraduates helping her. So the three of our projects, the goal is less to advance our projects and more for all of us to experiment with what can AI do? What are the possibilities? We have corpora. Let's sort of mess around and see what we can find. um you're right i can read the sources that i'm going to analyze for my dissertation in mind so for me it's less of do i need an ai to read them for me i'm

[48:01] not going to be analyzing the text of the german ones and the latin ones like i'm i'm going to analyze the text of the ones that i can read because i'm mostly focused on an italian context and i speak italian um but i'm transcribing the metadata in other languages, which I can manage. That's okay. It would be so helpful for scale. If my goal was to transcribe each and every one of these, I think maybe I might use something like Transkribus, which I have used and others on our team have used. But, you know, that's not quite the goal for my project. So, yeah, I guess that's where I am on that. Do you want to add anything, Ashley? I mean, we did try Transkribus, and I don't know exactly why the results weren't what we wanted, but that was the first thing we tried. I mean, we really wanted actually Transkribus to work because of, that's insane, like there's good ethical reasons to use it. It's built for what we're doing. We really wanted it to work, and it was really, I

[49:04] mean, And the project scope started with three tools that were all in handwriting recognition like Transkribus. So we really didn't want to move beyond that, but it just it didn't work well for these materials. Yeah, I remember when using my materials on Transkribus, which I think is one of the first things we did. Colleague Elizabeth, who was doing it, the formatting became very unworthy very quickly. it was comparable to like doing the transcription yourself um in that the the models for italian language for my period like just weren't as robust as some of the other ones as cosper had a bit so yeah but but we thought about it a lot um i don't remember it's been several months i'm too sorry no no no um i think that was one of the challenges though like was that um in the release of the models, it was, we were like, whoa, why aren't the results what we want them to be? And then we looked at how it was trained and it

[50:06] was like an, it hadn't been updated in a while. There's a lot of German, I think. Yeah. It was a lot better represented. So it may, it's one of those things, you know, may work a lot better now, but I think by the time we were, we were testing, it was like an older, yeah. Yeah. Your question. Yeah, this is really, interesting interesting interesting questions that have worked on similar issues um and it's been other contexts number one um wondering if folks could say a little bit more i heard like various kinds of confidence metrics i'm just wondering like what empirically those work right like what actually we use models this is kind of low or big confidence like what does that what does that mean one of that mean. Another question, I mean, and maybe this kind of follows on the point in the previous question, is it seems like I could be wrong, I could have missed something that is kind of the option of transgribence, which is not using kind of big LLMs or kind of LLMs potentially with instructions.

[51:08] And I'm wondering to what extent you also explored, like that's wrong as far as valid but towards you you played with fine tuning right because you have 100 actually taking an llm and then doing something training and whether that's accurate some of those third question i'm sorry i have so many questions um this is for the i'm haley haley this is for haley so in terms of the really big um the reluctance to use uh automated transcriptions, but it's a much larger data set. I'm wondering if you've thought about things, other questions that you could use those transcriptions for where the kinds of errors that you know would get into that result could also be where you could obviate some of the concerns around that. So not maybe using them for a close reading analysis, maybe not even for network analysis as much, uh more general you know topic modeling various kinds of uh word embed right where

[52:12] like a couple of little errors might come out in the wash a little bit and knowing that this is these methods have problems this results have problems but lets me make statements about a larger corpus in ways that then i can bring together with the close reading i'm doing right there are other ways that if you know that there's going to be error in the results there are other ways to get around be useful for sociology absolutely those are three awesome questions um i'm going to ask you to repeat them and then i'll repeat them into the microphone because i just want to make sure i get them right the first one the first one um is confidence metrics what what do those actually mean so the first one are what are the conf the confidence metrics we use what does that actually mean the second is whether you consider an addition um fine tuning models not necessarily transcribe is using LLMs but fine-tuning them with these kind of ground troops that folks have produced. Yep, second one is about fine-tuning the model.

[53:12] Have we done any of that? Have we considered it? What's the experience like? And then the third is whether if you've got model output that you know has errors, are there other kinds of analytic methods you could use that could produce results that are useful for your scholarship despite those errors, right? approaches that could still be useful. Totally. Okay. So third question is about if you have a large corpora and you have all of these transcriptions that maybe aren't perfectly accurate, can you still find some use in that for other scholarly findings? So those are really fantastic questions. Do you guys want to take any of them? I mean, the first one, yeah, I think that's, that's part of the toolkit that Cody developed. So Cody would really be the best to answer that. If you don't mind leaving your email, I can follow up with that question, but they'll respond in a better way. Yeah, unfortunately only some of us will be here.

[54:13] So I can answer a little bit on word and character level recognition. It's like the percentage of what was transcribed is word level or character level accurate against the ground truth. Does that make sense? Okay. And do you want to talk about? Fine tuning? Yeah. I mean, fine tuning we did do on a cross. fine-tuning personally but yeah if you guys want to talk about it well again it's at least in my project it came down to um getting to understand kind of proper nouns like contextual pieces like this is a caribbean writer like the word curacao and it not being greased like that's something that you know kind of had to uh be included as adjustments to the to the toolkit to the rule book go ahead oh i mean so after you've done one process actually doing like yeah training yeah yeah so that was that was done over the uh process of several months um to kind of get to where we we got to in our in our

[55:16] results yeah um and then uh on my own i did a lot of fine tuning where i I would put, you know, one particular handwriting in and I mostly use chat GPT for this. So this isn't representative of all the models or what one could do. But I would put one in. I'd say, all right, give me your best shot. It would give me its result. I would say, OK, here's my ground truth. Now compare, map what was right and what was wrong, what the differences are. And I'd refine it. And I kept doing that over and over and over. Really great for one particular hand. And then you go to the next hand and you find that it's overtuned. Um, the process also, I just found, I, I feel like it, it's, it was really valuable. And if I had one handwriting, I feel like I would have, I would have probably gone with that route. But I also, it raised a lot of, um, apprenticeship questions for me because I, I had a lot of unpaid labor opportunities as an undergrad that I put on my CV and helped me get to grad school and where I am today. and it made me think, you know, am I allowed to hire an

[56:19] undergrad? Like, am I allowed to, like, enlist an undergrad and not pay them? Like, where are the rules? Where are the ethics? And what does it mean to give that opportunity to a chatbot and not to an undergrad? So, I don't have any good answers for any of that, but it's a question I'm thinking a lot about. And then to your question of if I have all of these transcriptions and even if they're not perfectly accurate, can I do other things with it? I think a different scholar totally could. I think for the purposes of my project, I'm trained as a micro-historian, and I also do, yeah, the Italian tradition, of course, and my advisor's a micro-historian, and I love micro-history, but I love the changing scales and zooming in really close and then zooming really far away and valuing both of those. So for me, because I do really want a lot of my work to be a close reading, it's not something that I'm comfortable with, but yeah, absolutely. I think there are lots of ways creative, smarter people than me could totally find. The other thing on Laurel Bestock's project is we said, you know, the numbers were

[57:20] the problem there. And I asked this question, we were going to have more of us come, but I asked this question to Laurel Bestock this morning and I was like, you know, what, what happened with the numbers? We tried different things. And she said, actually we got pretty good after doing a lot of fine tuning, but it took a ton of work. And part of that is because the notebooks vary so widely depending on who the person was that was writing the triangles and the circles. And so it's hard in some ways to fine tune that because of the variability, which I think Haley's mentioning there. But then again, it still raises those questions of does it make the scholarly process richer? And I think for Laurel, that was still a very open question because she has a student that works with us on the project that checks all of these numbers because if they are wrong, that's really, really bad. She doesn't, Laurel then doesn't want to publish, you know, the years and years of these records with inaccurate data, essentially.

[58:21] So it just really changes the graduate student's role then. And so it became, I guess, like a philosophical question, much like Haley's, which is, what does this do to our research process? Do we like this? Does it make it richer? So the fine tuning in that case, I guess, did kind of work after a lot of work, but it still left some uncertain feelings, I guess, or we're not sure how to move forward, I guess, with the results. so yeah we'll do one more question like i obviously realized here and um so i have like questions but i also i guess like the question for you is like do you think i mean we're talking about like what is is this enriching the process but like is your hope that you're like i just really it's totally valid to be like i just want to be able to put it in the ai and have it do everything for me and just kind of like be done and not have to spend all this time doing these like kind of open to the idea of like hey you know if we did a little bit extra here like i think there's stuff like non-ai computer vision stuff that

[59:26] could maybe start to solve some of these problems and become more tractable than at the ai level but like is that is that that's like is that like too much work and you really want to do the one-shot approach or like what are you you know you're like oh we're open to ways to like maybe we can try it you know and again cody's not here to tell us what he tried but yeah yeah they tried a lot of things um to do it did we try um like messing around with the input images or just with the llm prompt and google books i believe they just worked on the prompting um the images are ones that each of us took ourselves from the archives okay and i think there's a lot you could do to the images before putting them into an ai that could make some of this go better awesome that's great but yeah in terms of contrast or making the text go a little more. Yeah. I'm curious, like, I've worked with it a little bit more before the models got, the visual, the vision models got really good in the last, like, six to eight months. But, like, it's also just, like, easier, like, it's, like, low stakes for, like,

[1:00:28] something like Kodi probably to try. Like, with the archaeological ones, I have no idea maybe the vision models are good enough, but you could probably remove, they look like they were on grid paper. Try removing the grid. Hmm. Thank you.