Abstract

Louis Hyman argues that artificial intelligence (AI) can restore trust in historical research when language models are used to write inspectable, reproducible code and not to answer questions directly, and he describes the model as an overenthusiastic intern and not an oracle. Drawing on his Johns Hopkins course Introduction to Computational History, he shows how students learn to read Python and to judge when a number is too clean to be true, as with the zero books recorded for California in the 1850 census. In his research he used agentic coding to find a little over 1,100 transactions by the slave trader John McDonogh in New Orleans notarial records, and he describes fine-tuning an open-weight Qwen model with low-rank adaptation (LoRA) on railroad forms. Questions cover digitization and archive funding, and how accuracy is measured against hand-corrected ground truth.

Chapters

Transcript

AI-generated transcript. It was produced from the recording by speech recognition and cleaned up with AI. It has not been edited by the speaker and may contain errors. The captions on YouTube are AI-generated in the same way. Each timestamp links to that moment in the video.

[0:05] Hello everyone, thank you for coming. One of our presenters grew sick, so unfortunately you have to make do with me talking about trust and teaching and a little bit of research. I'm going to show you an image that my colleague and friend Ana and I know very well. It's this image. And it's an image that we discovered together in the B&O archives. What is the B&O archives now called? National Museum of Railroad History and Innovation. Okay, the B&O archives. And the bosses will be really mad about this. But this is a very typical kind of document that you can find there if you are looking at 16, I think it's 16 million railroad worker documents, right? And this is the kind of image that document that so many of us have found over the

[1:05] years and just thrown up our hands and didn't know anything to do with, right? If you talk to my colleague, Kaylee Rosenthal, she says that about 60% of all archives are business ledgers of various kinds or bureaucratic forms. And this kind of thin information to do anything with. If you look at this, it's a statement from a hospital of an x-ray for a left ankle. And when we first looked at this, we were talking about, oh, we can transcribe this document, right? And we could, but that's not actually what's interesting about using AI for this, not just simply the transcription, but the categorization. And this is where I think we see something really revolutionary going on of the ability to not just transcribe, but to read and to organize. And this is in a file format called JSON, which even if you've never seen it before in your life, it makes perfect sense to you.

[2:06] You can see, right? Have you seen this before, Alan? Okay, so who's never seen JSON before? Great, excellent. What does this say here? Different categories of information. And they're all organized, right? Nice and easy for you to read. It's a human readable format. was when I really got excited about what it meant to do research with AI. To think about it as something that could deliver the bureaucracy into a way that we could finally make sense of. Now, this part is very exciting, but there's also been a wild misreading of what AI can and can't do. The most important thing to realize, I think, and I always say this when I'm talking to people, is that AI is not an oracle. It's an overenthusiastic intern. I think there's two spots up here. You know, it is this guy, and it's always a guy, who really wants to please you, who wants to read all the papers,

[3:08] also is kind of a mess and dumps coffee all over himself, right? And this is exactly why so many of us are dubious about AI, that we approach it like some kind of super Google, as if Google were itself an Oracle. I'm sorry. Let me move this stuff off here. Okay. You know, and this is entirely the wrong way to think about it, unless you're looking for, like, recipes or something like this. And why is this? Because when you ask a machine a question, it will always generate an answer. And sometimes it gives wrong answers. Then we don't trust it anymore. What is the right way to think about this? How can we make use of this intern, this research assistant, which we've all had, and we've enjoyed the use of interacting with young people who are enthusiastic, but somehow aren't able to fully understand what they're reading or

[4:09] make sense of what's going on, and yet we still value their work, we still value their contribution. How do we make use of these? In fact, how do we make use of 10,000 of these? of 100,000 of these, a million of these. And I think the important way, something we've heard again and again to the point now where it's a little repetitive probably now that I'm later in the conference, the ability to inspect the code, the ability to see how the information comes about. Now, you might think I would start this conversation by talking about my research, and a little bit I have, but mostly I want to talk about my class. This is mostly a talk I want to talk about my class, computational history, which is my pride and joy. Its syllabus is in the Discord if you want to look at it. And it's very exciting because they have no idea what I'm doing to them. They are signing up for one class and they're actually getting about four classes at a time. They are getting a class on statistics, actually

[5:11] econometrics to be quite honest. they're getting a class on computer programming about two semesters maybe two and a half semesters of computer programming and then also a whole history of data class at the same time and you say how is that possible Louis why would you do first off why would you do that to someone and the answer is because they chose it so they're up to they have agency in this choice But I think what's important about it is that it speaks to how we are able not to lower our standards with the use of AI, but actually raise up our standards. to actually push people towards thinking at the high level kinds of thoughts and ideas that they are able to deal with when they can push aside all the nitpicky things, like where is the semicolon in my code, to focus on the big ideas. And so by the end of the semester, and they don't know this, is that when I wrote

[6:15] my dissertation, I spent a lot of time working with the Survey of Consumer Finances. It's a data set you can download on ICPSR. It was a very interesting national survey of consumer finances every few years to the middle of the 20th century. It took me like six months to organize a data dictionary to figure out what kind of analysis I wanted to do. It was all in an ASCII file and a PDF. And that's all you can download, right? usually download. I give them two weeks to come up with a research paper. And when they have a research paper to do it, then we draft it. But they have to turn in the Python code they use to analyze it. The Python code has to generate every chart, every major finding, and it has to run when I hit play or they fail. They also have to write a paper that analyzes this.

[7:15] They have to do it together in a team of about four people. And this data is dirty data. This data is unclean data. You guys have all worked with data before. We've all enjoyed the pleasure of finding those typos, the things that make no sense. They don't get that in any other class. And so I have students semester after semester writing me from their internships or their consultancies or their investment banks because I encourage all of them to go into investment banking and donate to universities. they don't always listen. But I think what's important there is they say, Professor, you taught me how to clean data. You wouldn't believe it, but I had to do that on my very first day on the job. And I'm like, I do believe it. It's the first thing you all have to do when you work with data. Actually, in their other classes, they learn ever more esoteric math. And in my class, they learn beginning to end, how you go from data to argument

[8:15] And that is actually what history is amazing at doing, right? This is all that we're really good at, taking a set of information, making an analysis, and making it into a story. And when I talk to them, you know, you might think, oh, Lou, you're so naive, and immediately I am. These are just STEM students. Your class is just like a gut class for them. So I talked to one of my STEM students, and I say, my first time I taught this class, they're like, oh, this class, this class must be super easy for you. And he's like, what are you talking about? This is my hardest class. But the coding is so easy for you. You're a computer science major at a relatively well-accredited university. This must be so easy for you. And he says, no, this is my hardest class. And I go, why is this your hardest class? He's like, you don't even get it, do you? He's like, very accusative at this point. like why just back back up and then and he was like he was like wow you make me come up with my own questions

[9:18] you make me clean my own data you make me figure out which math i need to use to fix answer those questions you make me write the code and you make me write it right like write a essay the longest essay maybe i've written in college and you make me work with all these other people. This is the hardest thing I've ever done. I'm like, good. Good. And they are able to do it. Otherwise, they fail. And so for the first three assignments of the class, because it's a history class, and a lot of these STEM students can't hack it in history class, it's pass-fail. And they get to do it over and over until they get it right. So it takes away all that anxiety about They're not going to be able to do this. It's pass-fail. And they do it over and over until they get the assignments right. You can look on the syllabus to see what those assignments look like. But basically, the very first few assignments are all figuring out how to use a

[10:19] spreadsheet, figuring out how to write their first experiments in code, figuring out how to do basic data analysis, how to think about data. How do you do this? How do you teach them? And we've heard actually differences of opinions on this. This is my opinion on it. that students need to learn independent thinking, questioning, narrative, and data discernment. You know, what is a reasonable number or not? So in the first, I'm really feeling pinned up back here by this image. I'm sorry for those online, but I can't stand behind this counter. I'm like a caged jaguar or something like this. And I just can't have to keep moving. So in the very first couple weeks of class, I think this is week two, we look at the census of 1850. Now, as you all remember from your recent readings of the census of 1850, there's extraordinary data on books and libraries, right? Books and libraries. And so if you look at the census of 1850, you look at it, and I say to you, my very capable students, I say, how many books are in

[11:25] California? And the student raises his hand. And he goes, professor, there were zero books in California. because they were zero bucks zero it's a nice zero on that spreadsheet of the census of 1850 and he looks pleased and i go do you think there were zero books in california i have as much contempt for california as the next man i loathe hiking and wine drinking so i don't it's not a natural place for me. But they then realized, I'm like, why do you think there's a zero in that data set? They go, do you really think there's zero books? And they go, no. And all of you are like, of course not, right? Of course, this is like, Louis, you're not being very clever here. Part of my point is I'm not being clever. This is also something they've never been taught. they've never thought of before, that the data sets they get might not be, where do they come from? What are they about? This is something you all know.

[12:29] And then I say, well, where else are there zeros? In the Wyoming Territory, the Oklahoma Territory. What's the number of libraries? Oh, how should we think about libraries relative to the number of, why are books rounded in this way or not? Conversation about data and data discernment and how to tell when looking at numbers simply whether or not they're slop or not. And this is actually quite interesting, I think, because this is something they're not taught in any of their fancy math classes. And as you know, if you can't know whether a zero is reasonable or not reasonable, all that fancy math falls apart. Right? This is something we can teach and we have to teach. They're not getting it anywhere else in the university. They're not getting it in any of their math classes. when I was in grad school I had to take two foreign languages did you guys have to do foreign languages yeah yeah I took German from like the middle middle school with Frau Nagel she was really really nice I got to high school with Herr

[13:31] Leidenwaden he was less so and I was super bad at German and I got to grad I went to college and I had to take a placement test I failed that test got put in the first semester of German I'd taken it like my whole life I went to grad school Failed that test. I took four semesters in college. Still failed the grad test. I was like, I'm never taking German ever again. This is the devil's tongue. I guess it's Teufelsprache. I don't know what the word is for that in German exactly. But I got this book, French for Reading, which many of you may have also enjoyed. And it is an amazing book. Because unlike my German books in high school, it didn't try to make me learn how to speak German to buy pants at the mall. I don't want to pie pants at the mall in English. I don't want to do it in German for sure. And so when I'm seeing this, I'm reading French for reading. I am learning about French philosophy in French. It's very exciting. Six weeks later, I passed my French test. That's why I have a PhD. That's why I'm here with you today.

[14:33] Eventually, I passed my German test, too. That's another story. But what was exciting to me was, like, I can't write French. I can't speak French, right? But I can read French. As you'll see in a minute, there's things I'm reading in French for work. And so as I'm thinking about what is it to teach coding, I don't think we need to actually teach students how to write code. I don't think they need to be able to speak Python. I think they need to be able to read Python. They need to look at it and say, oh, I understand what's going on here. Is this a reasonable kind of stepwise procedure? Does this make sense? enough to inspect the code, to inspect the data pipeline, to understand what is going on, what are the operations. And so when I'm teaching, I'm actually teaching at these two simultaneous levels. On the first hand, they are learning the most basic things about coding. On the other hand, I'm also teaching them much more advanced ideas from software design, like abstraction and encapsulation, data pipelines, data contracts,

[15:35] thinking about the ways in which you need to design the relationships of data your data set and the analysis that you do. And you can do that because the AI will take care of the technical stuff, even as you need to teach students the more, I wouldn't say more advanced, but things you do, more abstract things. The things that are actually more human, more intellectual, thinking about those higher level issues. And so these kinds of things that I'm talking about are things that are not technical skills, but humanist ones. How should the numbers be? How can I think carefully about them? How can I wrest control of the discernment of the step of verification from the machine back to the person? During my class, I often show this slide, which is a way I think about how to do data analysis. So every time you have a data set, you want to look at the naive version. Assume all the numbers are right. Assume there are zero books in California.

[16:40] Get your head around that. Then you move to a critical stance. Then assume no number in here is real. It's all an expression of the power of institutions. This is pure federal power over territory and imperialism in the 19th century. Cool. That's a very important step. Okay, now that I know there's a zero in an empire, how can I make sense of this now? What can I still do with this number? is this useful to me as a reader of data? What kinds of calculations are possible or not possible at this step? And I encourage my students not to be entangled in one particular step, but to cycle through these moments, cycle through this kind of approach. And they're uncomfortable with this place, the incomplete but still useful sense of numbers, that numbers are useful but incomplete, and that we can still do calculations with them. We just have to not be naive about it.

[17:43] I want to show you a few different ways I teach math, because I know you're really here for that. I'm not going to go through all the math, but I want to show you a sense of what it's like to teach the sort of intuition behind math. So for me, I have a math degree as well as a history degree, my math degree is what they call useless because it was mostly taught by these ex -Sovietists that just mumbled in front of my classroom at Columbia at, no offense, Matt, and, you know, where I got the best math education at Columbia was actually in my econ classes, where you would have these math-anxious economists who were also strangely confident and had great here, but they would always sort of talk about the intuition of the math all the time. And that's exactly where we can still do that as historians. So for instance, when I teach standard deviation, I really speak to the anxiety of the people in the room. I say, why would I want the square root of something I already don't get, which is the square root of sigma squared, the square root of

[18:46] the variance, right? And then you just walk through it, and you're like, I don't know what that means. I'm very open about my own ignorance, right? I don't know what a variance of 70 means, but I know what a standard deviation of 8.36 is. And then I go through, you know, how to use standard deviation in this very practical way. It looks like this. This is not how I was taught statistics, by the way. So awful. Proofs, you know. And then just try to get them this intuition about how stuff works. And to do this, to be able to teach them this level of math, this level of programming, is a four-step process. a primer called LinkedIn Learning, which is like the corporate high production value version of YouTube. It's usually available on your university website. And then I lecture about it. And then I have them do an exercise in section or in class. And then I have them do a homework assignment. And so by having four touches in this process, we can get through a lot of material. And if they don't understand something, it surfaces pretty rapidly.

[19:49] Focusing on the intuition behind the math, instead of just proving things all the time, lets people have a clear sense of why things are. We all have a sense of what a yes or no is, an on-off switch. This is a way you can motivate, for instance, again, if we don't have time to go through all this, you can motivate why a normal distribution emerges from simple ideas like toggle switches and dice. And so things that look like a conspiracy, the idea of a normal distribution or as a social construct. You're like, actually, if you believe in toggle switches, this is how this number comes into being, right? And then you can show how it applies. So there are many things in the world that are normal. There are many important things that are not, like wealth distributions. And then we can talk about how you then can make sense of them using various transformations. So it's all very practical kinds of stuff in the class, thinking about how to use data, manipulate data. And again, when I'm teaching about coding, I focus on these high-level

[20:54] abstractions, literally abstraction, which is the idea that I don't need to know what's inside the black box that is the mysterious computer. Input and the output, right? And that you can't mess with what's inside it. And they say, Professor, but what if I want to learn how that works? literally a whole school right over there. You can go take classes on how to build computers from the ground up. But I use my computer every day, and I have no idea how to build a motherboard. We're all using levels of abstraction. We're all using levels of encapsulation. And I have colleagues in the stats department or mathematicians who are like, Louis, you should not be, you don't belong teaching this. You should not be teaching this. You need to teach them set theory before you let them learn probability, before you let them do. I'm like, no, I don't. I absolutely do not need to teach them set theory. Did you learn set theory before you learned arithmetic? Did you have to have a theory of numbers to motivate plus and minus? No. You learn what you need to know for the kinds of questions that you have.

[21:55] You develop the tools you need to get the answers that you want. And in doing so, I can teach them things they would not encounter for semester after semester in their classrooms, like how to use APIs. The other day I was meeting with my TAs and this very generous colleague who's trying to help me tutor my TAs and how to teach this stuff. And she asked my colleague who teaches the Gateway Computing class, well, when do you teach APIs and how to do IO and manipulate data files in computers? When do you teach that in the first semester of coding? You know what his answer was? We don't. When do you teach it? The second semester? No. When do you teach it? It's like, maybe never. And yet the ability to manipulate files in your computer, the ability to interact with things like Google Maps to geolocate things in the cloud, the ability to use APIs, is actually what makes your computer incredibly

[22:55] powerful. And in the era of AI, this is easier than ever before. If you try to do this before AI, it was incredibly cumbersome. Now with AI, it's incredibly easy to write an API request and get all the power of the cloud into your computer. So, in saying all this, to just remind us of what we do know, and what is useful to teach, in the sense that AI doesn't know what questions to ask, and it certainly doesn't offer them any ability to develop numeracy or literacy. Being able to understand conceptually what kinds of numbers to use, reasonable or not. And then also the sense of how do I make sense of a document? How do I make sense of the meaning of a document? And how am I going to put these two things together? And few things are as practical, I think, in the 21st century as the ability to

[23:58] discern what is or is not slop. And that's exactly what we teach. It's exactly what we all do on a daily basis. and nothing I said today except for maybe the standard deviation is outside what you all could teach better than me this afternoon. A student asked me recently when I was teaching this, she said, well, why don't I let the AI do it for me? I said, that's a really good question. And I asked her, like, what are you doing right now? Because we also play a lot in class. We talk a lot about play and my students make games in class work. And she's like, oh, I'm making an analysis program for my dad's business. He's a small business. It was like doing revenue and cost. I'm like, cool, cool, cool. So why don't you let the AI do that? And she's like, what do you mean? I'm like, oh, just let the AI analyze your dad's business and then tell your dad to do what the AI said. That's exactly right. She was like, my dad would never do that.

[25:01] I'm like, why not? She said, he wouldn't believe it. Lots at stake. Like, and that's why we don't let the AI do the work for us. We let the AI write the code for us to analyze the data that we can inspect. So once you have, if you add a middle step, the other step of writing the code, then you can inspect it so that the analysis is reproducible. You can see exactly what it did at every step because AIs lie. They are a people pleaser. they will hallucinate all the time. And it does it in a way that's very, very, very convincing. And so one of the ways that we can think about how AI could build more trust into our discipline is by thinking about what kinds of data pipelines we could be creating that are reproducible, inspectable. Not that we all have to do this all the time, but you could imagine a world where, like this is very common in social sciences, where they publish the raw data, they

[26:03] publish the analytic code, the paper come out of it directly. You can see every single thing that you can do. And 99% of us nowadays are taking image files from archives, they're taking pictures of, right? There's absolutely no reason why for our key parts of our analysis, we couldn't have the same level of reproducibility in our data pipelines. It would definitely be a hassle, no doubt, but it's also something that is not beyond the realm of imagination of what we could do with our pipelines. I'm going to talk a little bit about my work because I'm very excited about what this has meant for me just in the last year. Because I talked at the very beginning about bureaucracies and the sort of impossibility of analyzing bureaucracies. And so right now I'm inexplicably working on a few different projects. I'm working on a biography of a Baltimore slave trader named John McDonogh, who was also one of the principal architects of slavery in Louisiana and New Orleans in

[27:04] the early 19th century. I'm also working on, with Sam Backer, this project called Big Labor, Big Data, which is fun. This is the papers of Sam Gompers, the founder of the AFL. and I'm also working with I'm sorry I can't remember the whole title National Railroad formerly known as it's the B&O it's like the Prince of Museums like the artist formerly known as Prince yeah and so we have all these hundreds of thousands of records that we're working with there and in terms of research this is why I'm so excited that you can use the AI to do OCR but it can also help you do this automated wide reading that many people have talked about in other places, where it enables you to do the transcription, enables you to do the summaries, the entities, like what are the names of people and places, different kinds of topics. This is something that was done for generations by grad students

[28:06] who went through the Jefferson papers or the correspondence of Mr. Muckety-Muck or Dr. So-and-so and would write down what was in each letter and who is to inform, this can all be done by machine. So again, this is not something particularly novel. It's just something that I definitely can't afford to do. No one has the time to do anymore. Now, what is pretty interesting is not just what extending things that, you know, techniques and practices we've lost as we've lost massive funding, but also what's possible now that wasn't before. So it's not simply about doing old questions faster, now. So for instance, I want to trace the financial life of this monster named John McDonogh in Louisiana. I want to know every single time he bought a person. I want to know every single time he sold a person, but also when he bought and sold land or any other kinds of financial transaction. Every time this happened, there was a notary who signed a document.

[29:08] But the problem is in the records, it is indexed by what their creators care about, filed by seller, not by buyer. So how do I find this? Well, this summer I used agentic coding for the very first time to go through the notary archives of New Orleans. And I said to Claude, I want you to go through it this way and find every single thing written by John McDonogh, by John McDonogh. And I said, it will take forever. It will take six hours. a couple years ago my co-author and i Ken Lipartito tried to hire someone professionally to do this for us in in the nola archives and they said this cannot be done and this went through longhand french from 1805 finding all these transactions and suddenly you are able to inspect them and of course i went through them generated a little over 1100 transactions and this is the kind of thing you

[30:10] can do with agentic researchers that you could never do by yourselves. I personally would never spend roughly the next 10 years of my life sitting in New Orleans going through these documents, which is why even people who wrote their dissertations on this guy never did this. Similarly, he had a probate in 1850. The probate is about 850 pages is long. It's in long hand. It's actually a lovely secretary hand if you're fond of such things. Again, it would have been very, very challenging to spend the time to copy out every single thing in this entry by entry. People who have done work on it before never spent the time to do it. I OCR'd it in about an hour, and then I was able to write code using software that I built in about a half hour to then reconcile the counts of the values, the running count that's on the probate with every single entry in it along the way. I could build one-off software to analyze one document.

[31:17] This is the kind of thing that you can do with AI that I'm so excited about. And it gives you very important answers about the trajectory of his asset distribution by the end of his life. So through things like this, I'm able to reassemble the financial life of things that I think, and I would imagine you think, are quite important. Sort of financial practices and history of enslavement by one of the principal architects of American slavery in Louisiana. And this is the kind of thing that we could never do before. How does this work? Well, AI works by doing smaller steps. I'm sure we've seen this in other places. That you want to break it down. as Oracle, it's part of the AI as Oracle is not correct. You want to break the AI down into what would I let an undergrad do for me some step-by-step. You wouldn't have to do everything all at once. You want to break it down to different steps. In this case, this is a microfilm service card, the relief papers, things like this. I broke it down in, first I did the naive thing.

[32:17] We all do the naive thing. Lauren, did you do the naive thing? Oh, all the time, right? The naive thing is to do it in one shot. Why does it not work? And then you break it down to smaller and smaller steps, and you realize, oh, I get far fewer errors by breaking it into different steps, and also by using different models for each of the steps. And what's interesting about it is sometimes people say, hey, Lou, which model is the best model? And I go, I have no idea. Depends on what you're doing. They're like, really? I'm like, really? The best model, well, the best model out there will be not always the most powerful model, Claude Fable, whatever, may not be as good as a very cheap model you can run on your laptop for certain kinds of tasks in terms of the cost, time, quality trade-offs. So when you do this kind of work, you have to run experiments on whatever your source space is every single time. Now, as I say this, what's pretty exciting is that I am able to do this on my

[33:17] laptop without doing it in the cloud, without using ChatGPT or Claude. I use Claude all the time for certain kinds of things, but for these big bulk tasks, I often rely on open weight models. So open models are what makes it affordable for us to do these things at scale. And you can run them either on your university's cluster or you can run them locally on your laptop a lot of the time if they're within just maybe 10,000 or so. And what's exciting is, for me at least, is that a couple years ago, a few years ago when I started to get into this, it was actually very cumbersome to do these things. You actually had to type in the code, Python, even AI-assisted, it was very complicated, setting it up, making it work, reconciling everything. You can now, what's best, is to take Claude and say, Claude, I would like to use this open-weight model to create a two-step OCR process, run some experiments to

[34:20] see which is the best, and then give me the results and then do the work. And so Claude can run your open-weight models for you. And this allows you to have the best of both worlds in terms of being able to get the efficiency and cheapness of an open weight model while being the convenience of these frontier models that can oversee these processes. Claude can actually run this software for you on your university clusters. You just say, Claude, do this, and it does it for you after some steps. And these clusters are exactly the same kinds of GPUs that you'd be paying for enormous amounts of money for when you use ChatGPT in the cloud or when you use Claude in the cloud. A side pitch. I've written essays about this on the Substack, Computational History Substack. If you want to read more, I have all kinds of detailed ways on how to do

[35:22] this. This is something that many, me, Lauren, other people are very passionate about sharing with the history community, that it's so exciting the kinds of things that you can do at scale now. The last thing I want to emphasize is that we often have this idea that we never shared our work with other people. When we hear our conversations, they say, well, you can't trust an AI to do a thing. Would you trust your research assistant to write your book for you? You would not. We have systems for delegating the collection, verification of information. In fact, we've done this for generations. We have used this to organize massive teams of people. If you go back, you can look at the NEH grants. I've done this, and I encourage you to. Look at the NEH grants for all the big history projects, the presidential papers and what have you. There's page after page of structured information on how to organize teams of people to collect information. And I think, honestly, the best model for how to use

[36:24] AI is in those papers from the 80s, from the NEH, and how to think about to do this. So in this way, I'm actually very excited about AI insofar as it doesn't change how we do history. It extends how we do history with using new tools to ask more capacious questions. And I think that we should think of the machine as a reader, an incomplete reader, an imperfect reader, but a reader nonetheless that adds value to our workflows just like a research assistant would. But the code helps us keep it honest, disciplines the inferences that we're making. But I think we are still in the driver's seat. We are still the discerners. We are still the ones who confront the hallucination, making decisions about what is or is not real. And I think this is a wonderful time to be a historian if we tell the story about history correctly. That this is actually the most significant skill of

[37:28] the 21st century. Being able to see through the AI slop to what's really going on. Being able to help make decisions based on data and storytelling. skills that maybe we didn't have a moment ago, well, those don't even matter anymore. Whereas the skills that we do have, that we've cultivated and taught for generations, are the ones that are the most important right now. Thank you so much. Anybody have any questions? Yeah. I've been teaching birth year writing for a couple of years now and very frequently I have students come in and even though in uh I teach in North Carolina most of the people that come to uh my university are from this they are being taught through our sacred school education system um and Chromebooks and presumably about computer education and training but I still get a

[38:30] lot of not familiar with things like file directory systems whether it's in google drive or on their computer and i'm teaching some software and editing uh software is now and they are terrified of breaking things yeah what is it like in the classroom to be like okay you're gonna have to figure out how to run python code and here's what that looks like yeah you know how you set up a virtual environment and that kind of stuff like how do you make it still not terrifying and like on fire. Yeah. Well, obviously you just set a computer on fire and show there's no consequences. No, I think that's actually really true. I find that the hardest thing, it's not what older folks think is hard about computers. For older folks, they think what's hard is the Python. What's actually hard is where is my file? Like they keep everything is on the desktop or in the downloads and they don't even know what folders are. And I can teach them like sophisticated data types and

[39:33] algorithms. They have no problem with that. And they're like, why doesn't my programmer? I'm like, well, you're referencing a file that doesn't exist or it's on your friend's computer or like something, some complete nonsense. Like 90% of their, you've really tapped into a fury I have. Like this is really healthy for me. Thank you. I know 90% of all their errors are like, where's my file? So I think part of what it tells me is what I think is easy is sometimes hard. And what I think is hard is sometimes easy. And I think that is part of what's tricky about this exact moment, is that things we thought were very hard are now easy. And the things that to us are obvious, like there's obviously books in California, is actually super hard. And figuring out what that right balance is again, Yeah, they're very confused about virtual environments, where my files are. Once you get past that hump, they're good to go, but literally they don't have a sense of like why a file wouldn't be on their computer, literally on their computer if it's in their Google Drive in the cloud. Yeah, it's really hard.

[40:35] Yeah, yeah. Yeah. Yeah. while failing? Like, finding two standard deviations below the means of what you learn through it. Yeah, I mean, I think it's important to have low-stakes assessments early. And, oh, I'm sorry, yeah, you're telling me what I should have told everybody else. I really appreciate that. What helped, like, how do you find the failing student in a class like this? And the answer low-stake assessments. They're a pass-fail, but actually fail people. And when you fail them, you say, I am doing this because I want to support you as a teacher. You have not failed. This is just hard. I'm here to help you. This is hard. Of course it's hard, and I'm here to teach you. So the other day in my other class, my Social Theories of the Economy, we were

[41:38] reading volume three of Capital, nothing to do with this. they had turned in their papers and I gave out a bunch of C's in the era of AI you have to grade really hard on grading right, on writing and things like this right, and I said look, I gave you a C is not a normal grade at Johns Hopkins University right, and I said look, I gave you C's this is because I care and I want you to revise, I want you to resubmit and it's more work for me, I would rather give you A's and just watch Netflix but I want you to become good writers and thinkers and readers, right? I expect you to be better than the machine and I expect you to practice in my class so that you are better than the machine. It's my job as a teacher to make you better than the machine and that's actually the conversation we've had for 300 years, how to become more powerful as a human than a machine and this is something that if you speak to them in that honest language, speak to their fears like, do you want to be replaced? Then give me a terrible essay that I can help you revise

[42:38] and resubmit Oh, do you want to be replaced? I need to make sure you can understand the intuition that's going on here. And design your assignments that way. And design them in a way that a machine can't do them. And that's the way to do it. Yeah. I've been wanting to ask him about this session. Please do, yeah. And it's part of, I guess, my general kind of struggle with what we're doing with AI. of digital humanities remarks. We've been through the kind of same things again. It is really clear that for all sorts of reasons there's an enormous amount of energy and innovation and very exciting stuff going on around OCR. And that leaves me with the question that we asked all the way through DH. That means we're working with what's already digitized and AI is not helping us digitize material. And I guess I'm kind of concerned about that. You know I can think of things that I'd like to do with what I'm hearing at this

[43:41] conference which I can't do because that material is not digitized and in the environment which we exist I can't imagine how much more material we're going to digitize because of the cost and I'm just you know one of our real challenges always in particular is what we ended up working with the same stuff because that's what's digitized and there's a bunch more digitized now than 20 years ago when i started but i this seems like we're going to go down that road again where we can ask certain kinds of questions about certain kinds of people but not about the whole bath you know nobody can come up with whatever procedure there's a all of our stuff is digitized but i'm pretty certain it's still probably single digit yeah i mean i think i think there's two yeah there's two yeah i think that's a really important oh what what about digitization like in scanning how do we scan all that paper and right and to do

[44:43] generally new questions yeah um i think there's two parts to it i think there is a lot more that was digitized than it was 20 years ago um it's more accessible i'm always shocked what's digitized I also think that scanning technology has gotten better. So I bought a Czur scanner with the foot pedal thing, and it cost me, I think, $500, which is a lot of money for a scanner. But I can get through stuff like one a second or less when I'm going through paper and an archive. So I think that's part of it. I think also, you know, we've all gotten used to taking pictures of things. And what's shocking about it is that we did all kinds of experiments on this. Again, the experimental method. You need pretty low-quality pictures to be able to do this work. So I think for a long time, at least my impression, you probably know better than I do, my impression was in the aughts. There was a sense that you needed super high-resolution pictures to do stuff. That's absolutely not true. You need very small, much lower, yeah.

[45:49] I'm an archivist. but one of the best ways to get more stuff digitized is support archives. We are going to the same funding places that historians are going to. We are applying to the National Archives, to NEH. We need that money so that we can put that time and effort into those archival collections. If you want more stuff digitized, support archives. Support archives. Archives need money. I think there's also just creating pathways to digitization efforts. but yeah I mean I think obviously that yeah I think that's correct but I also think it's easier to digitize things than it was before yeah you know creating the kind of people so I well I mean hundreds of thousands of documents is more than I mean the some of these you know some of these are quite large but I got grants for some of them to do that kind of work I also think so a lot of collections there's more, I remember being there's more books on

[46:53] microfilm at Harvard than there's books in the library and microfilm is actually very, very affordable to automatically scan and so if I were thinking about places where I could go to town on stuff like this, it would be looking at microfilmed collections and thinking about how to transfer those in a way Actually, add on to this really quickly One thing I want to add preserve it for other people to use. I'm going to talk to my microphone. So when we go there as historians with our cameras and we use Tropy and we're ready to go with it, we're not actually making those digital scans available for anyone else, right? So it's really important to remember that the archives, the work the archives do help all of us collectively, right? So what we can recognize is that processing and making these materials digitally available from a database perspective is really easy now. We can create tools that Louis is talking about. The expensive more if we want to do what Stephen's asking, is we have to fund the digital part of archival work.

[47:55] And that also means that if we're not having people go into reading rooms and scan again and again and again, we make it available once, which is better for the materials themselves, right? I would say that we need, this is where the AHA could really step in, in a collaboration with archivist groups, is they say every time I go to an archive, I re-give my collection back to them. we tried that with Tropy and the archives would not get on board with it there's no disconnect between what we collect and what they want it was essentially unbridgeable that was our vision no i no and i think that well that makes perfect sense and i think that part of it is about education now so we we went to what's that organization the museum national no no no no no no no not the not the not the art not the museum formerly known as the bno railroad museum the the the big museum conference you went to association of museums and we were there at the american association museums anna was like actually all of you are incorrect about what we need to do and she like

[48:59] talked about file formats and there was like an audible hush in the room and they like could not believe it so i think there's an educational gap that we can straight with data, that maybe we should write a paper about this explaining that. Because I think that's actually a huge gap. And then we could do the thing that they wouldn't. I'm sorry, that must have been really frustrating. Yeah. Yeah. Okay. Yeah. Well, I was going to add one from the Discord. Oh, please. Do you think that the ease of analyzing tons of data in this way could be weaponized to find errors in other historians' work and then discredit them? Because we've seen this happening over the field. Could this be used to weaponize? I mean, I think that finding errors in each other's work is part of what we're about. I don't think that's a weapon. I think that's part of what's reasonable. And I think we as a community should be open to that. I make errors. Everyone here makes errors. I don't mind being told when I make an error. But I think that there's a difference between data fabrication.

[50:02] When people actively manipulate their data to make stuff up, bad. When people make an honest error, normal. And I think discerning those two different things is important. Yeah. Yeah. Did anyone see this? Somebody used Claude to fact check Paul Johnson modern times. And, you know, the Claude fable five, whatever, right? It said there are 800 factual errors. And, you know, as it turns out, it was wrong about maybe a hundred or more of them. The Economist said, you know, look at this, look at the systematic bias, like certain kinds of human rights violations are given more emphasis. Certain kinds of like numbers are inflated and it just tells you like what the agenda was behind this book. And so this wasn't by an academic historian, but it's a very well-known work. And I'm sure we're going to see a lot more of that. You know, and so what I tell my colleagues is like, who would you rather doing that to your book? Wouldn't you rather do that yourself first, like before you publish

[51:06] it? I write anything, I have Claude fact check it for me. I have Claude read it for factual errors, grammatical errors. I don't like Claude write for me, but I definitely have Claude read me. Yeah. Really enjoyed your presentation. Thank you. I'm going to ask a question about friction. Okay. Because a lot of this seems relatively frictionless. It's a matter of using different platforms. and after you verify that you make corrections and you're going to get to where you need to be where are the real shortcomings where do you see the where do you see the limitations of it and i asked in part because you know ben in our presentation was uh uh talking about ocr using uh ai for being doing ocr and was able to improve the output but didn't get when i saw the percentages of the accuracy. So you have some really difficult documents here.

[52:07] Did you get a point when you were using Internet AI or Internet AI where you were running really difficult documents through, it just was not giving you the quality and you realized there are still limitations? There's got to be friction in here somewhere. Oh, yeah. Talk about the friction. Can I talk about the friction where the dumb, dumb computer doesn't work? Yeah, that's the daily, okay? Yeah, the computer doesn't work most of the time. And part of why I'm not afraid of the coming AGI super digital god is Claude can't pay attention for 20 minutes. I'm like, go do a thing. And it spirals off in this weird rabbit hole. Or it gets something totally wrong. It fabricates. It tries to lie to me constantly. Right? I would never date Claude. So, I mean, this is unbelievable. Yeah, and so there's all kinds of friction where some of these documents of the lowest possible quality. Now in the sense that they're very challenging. This document here is one of the better ones, but it has tabular data.

[53:12] It has handwriting mixed with typed font at a different angle. It has weird things typed over it. It's a form. And so this is the most, when I talk to NLP people, they think this is a hard document because there's some handwriting It's not in a high, I mean, it's not sort of washed out a little bit. And they just can't believe this even can be done. And it can be done. You can see that railroad record right there. Pretty good. And partially it's because I fine-tuned the model. And so part of what's possible now is that, there's nothing to write on here. I forget. I think it's Qwen model. So what you can do in these models, again, which is really hard to do a couple years ago, So as you guys all know about how there are neural networks like this, and you put stuff in, lots of nonsense happens, it comes out the top. You can put another little layer at the top called a LoRA. And basically it just retweaks the numbers right at the last bit, and it'll let you

[54:16] fine-tune it very quickly for this. So everything else stays the same, and this little part gets tweaked. And you can do that with just a couple hundred examples. And so one of the things I did was I ran this through my system. I said, which one? And I was like, these are all crazy. They don't make any sense. Then I had Claude write me some software to correct it and then use that as training data to make this LoRA, this, I figure it out, stands for Rank Adapter, right? And this kind of fine-tuning, you think would have to be done in some supercomputer, right? laptop, well, it takes like four hours to do this. You run it overnight. It's super easy. You just ask Claude to do it for you now. It used to be very complicated. Like two summers ago, it was inexpensive. It was super complicated. You could not do it without like a lot of attention. But now it's very straightforward. So that's the kind of thing you can do. And I think you also need to accept error, right? And measure error and say, I'm not going to get everything.

[55:19] We're not going to get everything out of the archive. And that's okay. But to acknowledge where these errors come I mean, I think that's the thing. And measuring error is something we're not used to doing. We're not used to measuring false positives and false negatives. And Sam Backer and I have talked so much about this. This is something we're kind of obsessed with, about how historians are obsessed with false positives, like data fabrication. But we don't really care about false negatives. Like if we miss something, something doesn't exist in the archive, you're like, yeah, whatever. That's like how archives are. Of course it doesn't have everything, right? Why would I even care about that? Louis, why would you even care about that? raised for that. But that's like a whole obsession of a lot of social science disciplines, this trade-off between false negative and false positive. Something that we can talk more about, I think, as we go forward. I don't know. Does that answer your question? It does. Yeah. Did you guys not have good experience fine-tuning or did you fine-tune? Yeah. I mean, so we did. It was well before. I mean, this would have been impossible. So I just was curious with where we are now, what accuracy rate do you get even

[56:22] with doing that fine tuning? Do you get to like 90%? Yeah, like 90%. 90%, 95%. Yeah, yeah. And when you get the errors, it works pretty well. And with the software that I built for the museum, formerly known as the B&O Railroad Museum, it actually has the document on one side and it has all the data that's pulled out and organized on the other side OCR because no historian trusts that. I don't trust it. You don't trust it. And so I think what's interesting about it is we can build our own tools to follow our own workflows and our own beliefs and values. That's a good question though. Yeah. Maybe I'm missing something here a bit. How do you measure the false positives? Yeah. So how do you know that it's 95%? Yeah, because you make your own, you generate it, you look at it, then you go through the actual documents 100 times, and then you manually

[57:26] correct it. It takes like a couple hours. And then you have ground truth, and then you're able to measure it. So you just, and that's literally just, that's how you do it. It's kind of terrible and boring, but it's better than reading 500,000 letters or something. But you have to generate ground truth. is that a lot of the crowdsourcing movements of the early 21st century, like the Sam Gompers paper, they have thousands and thousands of ground truth documents. So we were able to train models very well on those ground truth documents. And I think that's true for a lot of humanities areas. I think we're out of time, unfortunately. So thank you so much for coming. Happy to keep chatting. Thank you.