Abstract

Three speakers consider what AI applied to digitized collections demands of archivists and historians. Richard Marciano introduces computational archival science, a research and teaching program at the University of Maryland, and an earlier computer-vision project on the Morgenthau Diaries. Linde Brocato explains how the Franklin D. Roosevelt (FDR) White House filed and indexed its correspondence, and Rajesh Gnanasekaran presents a proposed six-pass agentic workflow, ending in a Neo4j knowledge graph, for the FDR Presidential Library's gift correspondence. The Q&A turns to provenance and handwriting recognition.

Chapters

Transcript

AI-generated transcript. It was produced from the recording by speech recognition and cleaned up with AI. It has not been edited by the speaker and may contain errors. The captions on YouTube are AI-generated in the same way. Each timestamp links to that moment in the video.

[0:06] Welcome everyone. This is the Archives in the Age of AI session. The subtitle is What Cultural Institutions Need Historians to Know. that is not a historian, so that might be very presumptuous. But we hope to catch on a number of these issues. There are three of us, so this is kind of a three-part presentation. I'll start. I'm going to give you a little context on the spaces we work in, where I'm coming from, the particular aspects of artificial intelligence. to explore the context in which we're exploring them and using them. And we'll focus, I'll illustrate this with the number of history projects I'm working on. So I used history, started a professor in

[1:08] the School of Information Library Science at the University of Maryland. My dad-brown years ago was in computer science, that have been working for the last several decades with archivists and librarians and looking at visual cultural heritage at large. And so a particular topic I'm going to cover is something we started a decade ago that we're calling computational archival science. And it's something we're developing, learning, teaching programs, research programs using with students and professionals. And we'll illustrate that with several projects but the main project my colleagues will really get into, I'll start it, is something that was just funded to collaboration with the FDR Presidential Library in Hyde Park where we're looking at some of the historical

[2:11] records, communications between ordinary American citizens and the President's office and the government. And we're actually going to be there on Monday and Tuesday and start working. So we'll talk to you about that project because an earlier instantiation of that in 2021 looked at using computer vision and machine learning on related records and then our new project which just started which is funded through institute museum library services is going to look at generative AI and AI agents to try to accelerate the indexing exploration of these collections with the end goal of providing tools and interfaces for historians, students, and the public at large.

[3:13] So that's a little bit of space we're in. So there's three of us. I'll represent the space of archives. My colleague, Linde Brocato, who's a metadata librarian and Spanish medievalist historian among other things. We'll provide a lot more historical context on the value of really understanding history and records in order to do any kind of AI or proper AI. And then my colleague, who just completed his doctoral on a number of cultural projects. The latest one is also currently funded through IMLS. It's looking at conversational AI in the context of interfacing with historical collections that relate to the legacy of slavery and newspaper ads

[4:16] in the 1800s. And Rajesh is doing work with knowledge graphs and generative AI for creating and capturing historical and cultural context and trying to communicate that to these generic AI tools such that they are less likely to drift and make stuff up and kind of stay on message and can be more useful. So Rajesh also deploys artificial University of Maryland and so he's kind of on I was going to say on the dark side on the AI side rolling out these tools and so we have this this collaborate multi multi-domain collaboration which which is really essential I think we heard that in one of the presentations this morning I think all this stuff is moving so

[5:17] fast and it's so constantly changing and so overwhelming that none of us are really fully equipped to address this. The only way forward is to pool our points of view and sensitivities and experiences to try to make sense of what we could do, what we shouldn't do, where this is all going. So anyway, I'll try to go a little faster than this. So there is this moment, I think it's being framed that way. I remember a decade ago when people in the context of history were talking about the digital term. So now we're into the computational term. A little kudos to one of many journals that are out there. This is the journal, a Polish journal called Archeion. They're celebrating their 100th anniversary and they have, for those of you who are

[6:19] interested, they have a special issue on this topic of AI, modern technologies, historical records, and archives. There's a call for papers and communications on this. And I've been in touch with the wonderful editor, Dr. Kamila Pavelcik-Duram, who's hoping to roll out a whole series, a gradual online open journal series on topics that I think relate to. Speak a little louder. Yes, okay sorry, the audio person is coming right down to help sort out things. Thank you. Can you hear me better? Melt it out. So this is a space. I'm sorry? I think I can go that far. So this is the space we're in. It's framed it, coined it that way a decade ago, computational archival science.

[7:24] There's a link if you want more information on this. And the goal is to explore computational treatments of archival and cultural content at scale. And there's a long kind of involved working definition there, which I won't read, but it will be part of the slides. It's on this. So it's not, it's more than digital humanities. collections as data it's really even though there are overlaps it's really rethinking of the whole sort of archival processing workflow in the context of computational tools including AI, Gen AI or whatever the next you know iteration of technology might be moving forward so it's not about AI specifically but it's and automation and all the trappings that go with that.

[8:27] So another little plug for, it's still relevant, we've been doing this for a decade. We actually host a workshop at the IEEE Big Data conference by Desiree. So we kind of placed ourselves in a geeky computer science engineering setting to have a discussion space that also brings in history, archives, libraries, cultural. And it's one of the more successful, it's been one of the more successful workshops. We have typically 50, 60 attendees who come every year. This year, it's the 11th year we're doing this. in Phoenix, Arizona. We have a call for papers. So if you'd like to follow any of these links, you can see what the call for papers for 2026 is. The deadline, plenty of time, is early November.

[9:28] So if you're inspired, you've come up with interesting new ideas, plenty of time to write a short paper or proposal and join us either in person or virtually at this workshop. So we've been doing this for over 10 years. There's some 200 papers in these spaces including a lot of stuff on AI, Gen AI over the last few years and some 50, 60 workshops we've had and collaborators all over the world. These are some of the projects we've been working project that Rajesh is leading on harnessing generative AI to support exploration and discovery in library and archival collections. So there's some long acronym which comes out to Heritage AI and we're developing, we're working with several cultural spaces and looking at these kinds of solutions. These are two brand new projects that were

[10:32] just funded a few weeks ago. One is called Connect FDR. There's a link there. So it's unlocking FDR's correspondence with ordinary Americans using Gen AI and agentic AI. And a second one, and that's my colleague Mark Conrad, who's a former visual artist at the National Archives, who's working with us as well. And all three of us are part of the project. And then there's a second project with my colleague Jane Greenberg at Drexel University of Philadelphia called Computational Archival Science and Metadata in Practice. So we're going to be working in very small print with a number of cultural heritage organizations, the Library Company of Philadelphia, the Academy of Natural Sciences at Drexel University, the Free Library of Philadelphia, and Franklin D. Roosevelt Presidential Library, and organizing a year from now a national forum to discuss what we've learned,

[11:36] the possibilities, where this is going, what we can do, etc. And so we'll try to, there's a link to a website. We hope to have a number of events. If any of you are interested in these kinds of topics, please contact us. We'd love to work with you. So we're all part of this institute that was founded in 2020, just before COVID, unbeknownst to us, called the Advanced Information Collaboratory, where we're specifically looking at activities and objectives that resonate with this event this week, exploring the opportunities and challenges of so-called disruptive technology. for cultural heritage information. That includes AI, but it's not specifically about AI. And leveraging these kinds of technologies to unlock information in massive record spaces.

[12:37] So there's a notion of automation and scale in there. So this is a research network, so we have some 70, 80, it's growing colleagues and researchers all over the world who show up at our workshop and work with us and collaborate. Since I'm in an academic setting, I've incorporated a lot of this, especially in the last two years, in the curriculum itself of masters in library information science students. So there's a little link here, a little communication that just came out yesterday describing this, on how students learn with records they know intimately and really well. So I've been teaching for the last two years, it's a five course series on introduction to computational archival science. and coding for non-coders. So we use Jupyter notebooks and we have

[13:40] an approach that is modulated and adapted to humanities students with no technical background, certainly no coding background. And students are able to deliver really interesting projects in just a few weeks. So, and this wasn't taught before, so this is one of our interventions. The second one on visual curation and data science using archival materials. And then we have a series of other topics which is spatial representation and analysis of records and collections. And graph representations of historical content. And then the last one, which is described in this article, is a course on Gen A&I and large language models for library information science students working with collections and records and going down the running level.

[14:43] A couple more items and then I'll try to provide all the slides, but probably it's best that we hear from our colleagues. So all this work is very much grounded in the use of historical sources. We have this repository that we call LTAR because you have to come up with another acronym. archival research datasets. These are some of the projects we've been developing for a number of years. So I have a whole project on World War II, Japanese-American incarceration, working with the National Archives and community groups for over a decade. A project which is newer, which Rajesh has done a lot of work on, legacy of slavery records with the Maryland State Archives. This FDR project which connects to American responses to the Holocaust, looking

[15:46] at FDR records. And then some other projects have been going on for a long time. A project on the history of redlining through the Homeowners Loan Corporation in the 1930s under FDR. That's a partnership with the University of Richmond where we digitized maps and all the records and have over 10 ,000 neighborhoods with all the metadata and spatial data that you can incorporate in projects and build on. And then a more regional project looking at the history of urban renewal in the 60s and 70s in Asheville, North Carolina. So I've been working with community groups translating all the paper records into digital content and map content and I use, we use all these records and somewhat processed records in all these computational archival science courses.

[16:47] They're actually the foundation for developing computational thinking skills. real records with community partners. Lots of other projects that are infrastructure-based, social justice -based, cultural heritage, education, lots of papers coming out of this on all these different LTAR collections. These are older than quite a bit since then. These are all AI, Gen AI related and they look at these various collections and things like interactive interfaces, conversational interfaces to legacy of slavery records, transfer learning on legacy of slavery records, computer vision, machine learning with African American photo collections, etc.

[17:47] So lots of partnerships and interesting experimentation. In all our courses and everything we do we try not to emphasize the tools even though we use specific tools but we try to come up with concepts and ideas that students can design experiments around. So we have a computational thinking taxonomy or matrix that we use in order to integrate the use of ephemeral and fleeting and disappearing tools so that the conversations are more permanent. We've also developed a portal with cultural heritage case studies based on real records and computational experiments. It's called cases.umd.edu. It stands for Computational Archive Science Educational System. We use a lot of that in our classes and colleagues are starting to use that at the

[18:51] university. So this is kind of blending historical content and computational experiments and capturing those as interactive notebooks that you can share and run in the cloud or locally. I'll show you just a few more pictures, but this is kind of how we work. So this was a semester-long team -based project on World War II Japanese American incarceration records where we blended undergraduate students and graduate students from a number of programs, including students from history departments, from computer science departments, from library information science, and created heterogeneous teams where you had kind of representatives from all these different spaces so you could learn how to work together and contribute unique

[19:53] feedback without feeling like they had to become programmers or developers or turn into experts in other domains, which is not desirable. And I'll just end that comment with this. So here you have five different miscellaneous teams working on various aspects of these records, all communicating using this computational thinking vocabulary or matrix, using different tools they're able to relate their experimentation through this sort of metadata, meta layer that addresses computational aspects. So I will end, that's my segue to my colleagues, with the Morgenthau project and this is something we started in 2020. So the first part is this kind of machine learning experimentation

[20:54] and then my colleagues will talk about the current version which we're about to launch which is looking at agentic AI and why we're looking at agentic AI and they'll give you a little more context on what we should what we might want to do with AI what we shouldn't do and And then there's some takeaway messages that Linde will help deliver as well that relate to the title of this session of why are we doing all this, what's the point, and what really matters in the end. I thought that my colleague was a deep thinker and much more insightful than I am. I tried to deliver that message. But this is one of the collections we started. It's some almost 900 bound volumes that document the day-to-day business activities of Henry Morgenthau Jr. who was actually the Secretary of the Treasury

[21:58] from the early 30s to, I think, 1945. So this unique record with every single conversation, transcript of phone conversations, memos, memorandums of understanding, reports that's structured on an hour-to-hour chronological basis. and we're using that to ask bigger questions on the nature of what the government was doing, what they knew and the connection with the Holocaust. So what we're doing in this project, just a few slides, I promise I'll stop. We're taking the unique table of content from each of these near 900 volumes. By the way, this was supported by the wonderful staff of the FDR Presidential Library, Kirsten Carter, Bill Harris, and at the time, Abby Gondek.

[22:59] So we take the first pages, that's what you see in the top there, every volume has 5 to 20, it's not a book index, it's really a topic map of important issues. and they were binding these on a weekly basis and using them for the delivery, for the pursuit of policy and conducting business in the government. So this was an active internal database. It's a really unique and interesting project. and what you'll hear from Linde is that this was really incredibly innovative and significant. So the first thing we thought we would do is try to understand the indexing, cataloging, and search mechanisms that were actively used during the FDR administration and use AI to not negate those

[24:01] or reproduce them or recreate them, but to mimic them and recreate them in the computer so that we could then leverage other things. So we take these table of contents and we apply computer vision algorithms similar to what you see on the left there, detecting objects and cars, to detecting sections in those documents based on indentation. So we actually trained machine learning algorithms on a subset of those table of contents and then let it rip for across the entire collection. So what comes back are individual blocks that correspond to topics, important topics that that we use to index into all these reports in real time. And then we use a second machine learning phase where within each of these blocks

[25:01] we try to extract a number of features and those include headers, subtopics, the content associated with those subtopics, dates, book number references, page references and all that. So here you see how the computer vision algorithm takes a single one of those blocks, this is a very significant one, this is the War Refugee Board created in the beginning of 1944, and and breaks it down into all its constituent components and stores that simply as attribute-value pairs in a database. So it's a MongoDB flat attribute value page. And that allows us to then run additional, more modern, artificial intelligence processes that do more traditional things like

[26:04] NLP name entity recognition. So locate on top of that all the significant locations in these documents, names of organizations, names of people. So we're recreating the past and the processes that were used and we're augmenting that in this instance with a few entities. But sky's the limit. You could add on to this MongoDB database all kinds of other sources of information and then link that to other collections, which is what we want to see here. This is a crowdsourcing interface, very important, that takes the output of the machine learning processes and it's called the Morgenthau verifier and allows you to validate the AI generated content in a kind of crowd -sourced way, which is also essential. So we also created an API to this MongoDB database so that you can generate storytelling or write

[27:05] scripts invoking calls to the library that's built on top of the attribute value database so this is a Jupyter notebook that asks questions and then interrogates all the content that's been extracted through machine learning, augmented through NLP processes, and validate it through crowdsourcing. And you generate all kinds of visualizations and analytics around this collection of table of contents. So I'm going to stop here. If you're interested in any of this, we have a Google group on computational archival science. And then we have this annual meeting that's coming up in a call for proposals. So thank you for your attention.

[28:08] Thank you. I think I've had too much coffee today and so it has to be in fact bossier than I usually am, but he's in an exterior sort of way. So I am a metadata librarian, but by initial training, I am a rare book cataloger, but by initial training, I'm a medievalist in Spanish, largely working with Castilian literature centered on the 15th century, but that of course means that I have to do a lot of history besides. So my part of the presentation actually looks at precisely the kind of context that's important and useful. So, I mean, we all know that FDR was elected in the midst of the depression, and when it was absolutely clear the status quo was not working. And so the New Deal was essentially reworking the economy and the way the government worked. They needed systematic information and so the FDR White House, the FDR

[29:12] administration, set about finding ways to get coherent data and to make it accessible both internally and externally in a timely and coherent way. And that, what happened with the correspondence, the correspondence with the White House, the The Herbert Hoover White House had about 400 letters a day, and the FDR White House had about 4,000 letters a day in terms of correspondence, and that had to be handled. So they wanted to be able to lay hands on whatever was relevant and necessary as quickly as possible, and so they used cutting edge systems to do that. The FDR Presidential Library is the first of the presidential libraries and all in the many records were and they began transferring records almost immediately as you can imagine they would if they had that much correspondence coming in. So these are just some slides about the FDR

[30:16] Presidential Library in case anybody doesn't know. You guys are historians, not the staff at my library, so many of these, some of the factoids are probably not registered in your memory on a continual basis and not in mine either. But it's quite curious because, as you can see, FDR is who signed into existence the National Archives. talk about the National Archives as a building for almost a decade, and the point at which the National Archives were established as an archives and became an administration and an administrative unit, not just a building. So the plan was to include records of the era and of the FDR administration, which included, of course Eleanor Roosevelt's records as well. This is the only one for that. And there are over 17 million pages of documents, official and

[31:17] personal papers of the Roosevelt's and also the Morgenthau records and a whole bunch of other stuff that's super useful. All of these are what we would now call analog, right? They're paper, historically paper, they are not born digital. And when things are digitized, as we all know, the OCR underneath is not accurate. So this is the web page for the FDR presidential library. This is their finding aids are all digitized and this is the landing page for the research engine. But this is what Roosevelt had to say in November of 39 when the cornerstone was laid for the building. And the thing that's important is that he was pushing for public access and transparency with government records including all of the correspondence of the White House. So as he said about guiding the recovery of the nation and then of course guiding the nation through World War II,

[32:21] the information had to be made accessible and he stacked the executive administration with a whole bunch of hardworking people to gather information and make it accessible. These are a bunch of the agencies that were put in place and to the point that at the end of 34, in 34, there was this publication that was a guide to the government agencies that the New Deal had set up. And I have the highlighted box around that first chunk because the author, each of those subpoints is the same information focused from a different perspective. So basically there are five outlines of all of these new government agencies from five different standpoints, points of analysis, axes of analysis. And this is typical, right?

[33:22] Here's R.D.W. Connor, who was the first archivist for the United States, who ran and set up the National Archives. And this is what he had to say about the FDR library, which is quite curious because his first sentence I think is, gives us a huge As I would say in Spanish, Franklin D. Roosevelt is the nation's answer to the historian's prayer, right? He has a deep historical sensitivity and he set about therefore preserving all of the of the records that he could lay hands on, as he had done before. Mr. Connor, Professor Connor, is quite interesting in that he, in his article here on FDR visits the National Archives, which was published in 49, long after FDR had died, but he's recounting events of 34, 35,

[34:27] 36. he recounts how the president basically appointed him, came to appoint him, and how the president recalled that as the assistant secretary to the Navy, Connor had gathered records in North Carolina on naval history, revolutionary of the Revolutionary War, for an initiative that that Roosevelt had set in motion. And here the funny part is that Roosevelt said that he needs to appoint an archivist and asks about the pronunciation of the word archivist and then says he needs to be a good Democrat and he must have the endorsement of the historians of the country. And our friend Mr. Connor says, well, I can attest to the first, but I don't know about the second. And FDR says, well, I do know about the second, right? What that little vignette says to me is that FDR was

[35:28] reaching out and learning about everything he needed to know in order to make these sorts of informed appointments. And the curious part, and the reason I put this second paragraph is because then the president delivered a dissertation on the needs of archives, which to me gives a huge insight into FDR's own awareness of the importance of gathering evidence, gathering information, right? And then making it available. So in there, I think this is where I'm going to talk about this. That slide may be in the wrong place. The interesting thing about this where it says unbiased minds is that the outcry against Roosevelt's self-promotion, as it got called, seems to me, as Connor says here, to kind of miss the point because the

[36:31] inspiration of FDR doing that was knowing that the information needed to be gathered, preserved, and made accessible. And so the new approach that he wanted to take was based on, was essentially evidence-based, right? And so the evidence had to be gathered and the tools that the FDR administration used to gather them are these recently formed or nascent or developing fields that I list here. And all of these folks that get deployed by the executive branch and in all sorts of the projects of the New Deal are social workers, sociologists, historians, and so forth. There was in fact a WPA project going to places where historical records were held and were in danger and transcribing them. So they were paying people to go out and basically find ways to preserve and make records accessible. You can still find those publications.

[37:33] And I offered this, and I'm not going to talk about it long because there's not that much time to talk about it. This was published in April of 34. The FDR was inaugurated in March of 33. And you can see the Roosevelt year. And part of what I'm talking about is precisely what we're going to see here. So here's the title page and the sourcing of all the credits for the photos. Because the very next thing that happens is an index. And the index is in extremely deep detail. It's mostly named entities, places, and people. But at the same time, it's a deep index. And Mr. Pirates, who produced this, who was, in fact, an employee of the administration in the executive branch, basically says, picture book should not need prefaces. We had a production problem. If you find that there are discrepancies, let us know, because it was because of

[38:34] this production problem that we had. And if you'll notice in the bottom of the third paragraph, the last person he thanks is a woman, right? Without whose counsel he could not have undertaken the project. But if you look at the way the book moves, so here's the last page of the index facing page which is essentially one of those wonderful WPA photos. And then it's thematic, page by page, and it lays out what had happened in that year, leading up to the inauguration and then during the whole year. It does not spare the administration of criticism when things sort of fouled up. There's a section on trying to deliver the mail with military planes, and so there are like five photographs of crashed planes and the admission not work it was an experiment that failed and we got to figure out something else this was i thought super interesting because who knew that people in the in 33

[39:37] thought and 34 thought that the technocracy was going to save the country from the depression um i'm not sure i would want to be miss technocracy but there we have it so so here is as a professor of mine in graduate school used say you know whenever you publish something or give a talk you have you have heard it said which is lit review and then so the in terms of you have heard it said it promoted it's the PR administration promoted itself and made public its successes and that is certainly true but it also simply made information accessible, not all of it was self-praise eliminating things that failed. There's a lot of scholarship on that, as well as at the time in the 30s and 40s, a huge critique of what was perceived to be self-promotion on the part of

[40:39] the executive in this person and also the executive branch in terms of its power. But there is no deep accounting of precisely the that made that information accessible in a timely and coherent way both to create those those reports and the dissemination that went out to the public as well as to other branches of government and also made decision-making and policy-making possible and so the traces of that work right are what we are dealing with as we mobilize this project. So FDR's executive office made use of all of those then cutting-edge tools that I listed from those disciplines, from those fields of work, to find and deploy information. Oh, that's a duplicate of that slide. Sorry about that. Okay. So we worked with two sets of documents. You saw the Morgenthau Diaries that Richard talked about.

[41:40] And what we're working with now are the President's personal files. The Morgenthau Diaries have a chronological structure and they're a fairly closed set. Most of the names that appear in those are going to be well known and so name and named entity recognition can work really really well. The president's personal files are much a bit so much more open set and it's much more random so we have a different set of issues and problems. The other thing that happens, we're doing the same kind of work that FDR staff did at varying levels and we have to recuperate the building blocks and processes that created the complex information objects which I'll show you one in a minute. And if we don't do that, we're going to get multiple hits for each entity, each transaction. So these are basically some of the features of these documents, and now you get to see what you look

[42:45] like. Now, remember these are produced on paper. These are digitized. They are not even particularly clear modern type. These are typewriters from the 20s and 30s. I wanted to type on my mother's typewriter, which was patented, I want to say, in 1923. So I am familiar with these. So this is the cover of the file, and you can see there's information on the tab, right? And then you have, this is the first document in that file. And it is, in fact, a carbon copy, which means the letters are fuzzy. It doesn't make a difference to us. It does make a difference to AI and to OCR. But when you look more closely, I have put colored boxes around the different kinds of information. So we're calling a transaction. These are the GIF files people sent FDR stuff constantly. They sent him food. They sent him books. They sent him sketches made from

[43:46] photographs of him that appeared in media. And so what you're going to see on each of these, this is a cross -reference. abstract. Everything that's circled is a kind of information and the turquoise circles and the blue circles are where other copies of this are going to appear. The transaction group appears under the person's name or corporate name that sent it. If you are Smith and you sent a book, there's also a copy in the beads for the kind of gift you sent. Which is why we have to track this for one thing the cross referencing is all here also who typed it where what kind of file it's in where any cross references might be but those are often handwritten right so we get when you get CPPF those are the presidential personal files so that's a different correspondence file not the gift file right and you

[44:49] have p pf 9b this is where this is so under minor school which is on i grew up in birmingham alabama so i was quite startled to find this this cross-reference to a school from inslee alabama which is a kind of a neighborhood of birmingham um on the work on the on the workers side of Birmingham over by the steel mills so you get a whole bunch of different stuff that we have to track right first we have to figure out how to get the computer to see we have to figure out which of it is going to be necessary for us right which we're going to be able to abstract which we want to use this is an acknowledgement letter and the blue circle at the top left is around the staple which indicates a group this is stapled to the original letter often handwritten and if there are any like notes processing notes has to be given to somebody I don't know what the hell this is

[45:51] right those are also stapled inside which means that you wind up also with incomplete so you'll have a bigger document and then a smaller document and sorting that out is hard so this is essentially this is a summary not just sort of a cross-filing abstract and there the the information complexity is even deeper this is another acknowledgement of something that was this was a memorandum to the State Department because I think this is the one where a guy sent something that his wife had been his late wife had been working on he was living he was like from Ohio and living in Canada and sent this to the White House and then there was a customs fee and so it involved the State Department and the Executive Office and a whole bunch of people which means that this particular transaction group is actually like six or eight documents, right? These are interesting because when the White House referred stuff

[46:55] to another department they just did a mimeograph sheet of all of these are books and titles right and so they did as many of these sheets as there were places to file it right so this you would have eight ten twelve copies filed in different places so you begin to see our problem in terms of actually an accurate AI approach to indexing because we don't know what kind of document it whether this is where the main body of the information is gonna be, or if it's one of those cross-reference pieces, then we have like six copies somewhere scattered through the files. So I began, when I looked at this, because I do medieval literature and manuscripts and early print and all of that, I looked at those documents and I thought, no, wait, we can't do a simple. can't do a simple one-pass prompt here.

[47:58] There are too many layers of information here, all of it potentially significant. So I basically went through and grouped everything. The green and the blue are, those are document groups, which is why we started calling them transactions. And then the white ones, the white lines are all of those cross-referencing pieces. So So our point is basically that there are entailments to the complexities of these documents. And the entailments mean that there are many kinds of materials in these PDF files because there are many types of materials in each of the folders. We're going to have a problem because the original gift transaction is in reverse chronological order. So you have acknowledgement processing of the original. The abstracts here are memoranda, they are a list of materials transferred to other departments. All transactions appear at least twice in the files.

[49:03] Separate images of complete documents have to be separated. All pages of the PDF need to be accounted for, either grouped in transactions or characterized as to the kind of document they are. So that makes the entailments for training the AI, which is Rajesh's problem, not mine. I only give him the profits, that's all. Is that we need a controlled vocabulary for talking about this history, the old -fashioned field of diplomatics. Also, those emerging fields of records management and the management documents that the executive branch and the office of the chief of files produce. Because the abstracting and filing documents as well as the acknowledgement letters are rigidly structured, we can use that for training the AI. This is where the metadata part comes in. AI wants to summarize. We want to characterize. Metadata characterizes rather than summarizing.

[50:05] We have to decide what kind of balance, how we're going to go about that. and the AI is going to have to be prompted in stages. Rajesh has developed an agentic. So this is basically how I was working out kind of the metadata profile for this. The slides will be available also tidied up. And this was sort of how I imagined it, but that's because I am not a computer scientist. That's because I am a different kind of thinker. So our conclusion is we're using the work of this anonymous army of people who were taking librarianship, indexing and abstracting records management, and building systems so that the executive branch of the FDR government could find information, synthesize it, offer it to the public, and also use it to produce the

[51:06] kinds of reports that were there. created a level of accuracy that we cannot get at using AI tools. AI is not strong on accuracy. But the AI tools allow us to open a path to get to their accuracy. One thing that they do let us do, though, is what I call to be a kind of residual memory. We've all had the experience of saying, do you know where this is? And somebody says, yeah, I think I saw it in this other file over here, which, of course, does not have that in the name. that's not how it's indexed right and we do that also with the kind of research that we do in the sources that we work with and AI tools I think will let us get at that um and it just to answer what cultural institutions want historians to know digitized records are often complex and require complex AI approaches collections often document their own tools in production and that's important the archivists and curators of those collections are expert guides to those levels of complexity and AI helps analyze

[52:11] and represent that complexity also that complexity merits study I don't have enough life left to take up this project so you know find somebody working on early 20th century history because I think that this approach to information and information and public administration and policy development and in information access in organization is absolutely worth looking at and understanding in depth. I mean we're at another one of those moments where the status quo ante is not working and we're going to have to rebuild and there are tools here. So Rajesh. Thanks, Linde. Thanks, Richard. So, yeah, we wanted to, yeah, so as you heard from Linde, so basically the problem is complex, right? So with a list of documents, which is of different types. So we wanted to come up with proof of concept or proposal on how to handle these

[53:16] documents through AI, or using AI, right? So, here is a little picture on the different type of problems in terms of document types that we have. So, mixed inscription types, there's documents which are handwritten, which are typed, and there are also sort of what Linde mentioned is process notes. These are basically sticky notes that go on top of individual documents and they're all stapled together as a group, right? And we wanted to actually peel all these layers one by one and identify all the data that metadata basically so that we can characterize and provide, you know, grounded responses through agentic AI rather than, you know, made up or hallucinated responses or maybe having AI summarize it just by looking at these documents, right? And there are also cross references like Linde was talking about. So these were all on one piece of paper. So there's multiple information, facets of information present on a piece of paper

[54:19] and individually and also as a group. So and this was all tacit, right, in the sense the staff was actually doing it on the fly and they were doing things on the fly. So there was a method to the madness but then there was, it was not documented, just the method that they had followed, right. We do see the groups of documents filed into the documents, I mean the folders, but we know what went into the decision in 40 to get. So what we have proposed is basically a six-step or a six-pass agentic AI workflow. You can call it a pipeline, you know, or workflows basically. So what it does is like Linde and Richard was saying there were about 700 folders of this entire GIFs collection that, you know, included gifts from our letters and objects from you know common public to FDR's office right and we take each of these folders and we just experimented with one of the folder they are all numbered and you know alphabetically arranged so names will be like part of

[55:23] let's say for alphabet B then there'll be names also occurring with the objects as books and other stuff they're all arranged in that order so we have to kind of look at each of the page within if these are scanned now so you can look at each page and then start with identifying what type of document is through Linde's funnel. We need to know what is this? Is this an acknowledgement letter? Is this a gift letter? Is this a filing note or a process note? Is this a sticky note on top of something? Is this a carbon copy? So we have a, with Linde's amazing work you saw earlier with annotating and you know, marking all of this, we have some trained professional looking at it and then given us the list of document types to start with. We don't know if it kind of encompasses the entire list of documents, document types, but we have to start somewhere, right? So we use that, I mean, Linde's data and give it to a cloud which is through API and then kind of like

[56:24] have it identify what type of document it is given a specific prompt for that particular step, right? what document it is, then we go into the second step of grouping, right, because now we know what each page looks like, but we need to know what pages actually have to form together as a group, so we can do analysis later in terms of, you know, like making it a conversational interface or whatever, but it's grounded into the source that we are in the grouping there, right. And the group could be like individual, like Linde was saying, it's all in the reverse order, like acknowledgement letter that goes back to the gift sender and process notes, filing notes, cross-references and all the way back to the actual original letter. So if we group them together and we can have that sort of structure, then we can, now we have what pages these are, what document types these are and what groups they are forming into making it a transaction.

[57:26] And the third step would be, or the third pass would be to extracting all the useful information from it, right? All of this using generative AI, right? So all of this will have its own prompt or system instructions, if you want to call that way, and with additional information and context provided by Linde, so we'd make that an individual step. And in the third step, which is to extract all the metadata, which is what makes it, you know, classify the data instead of summarizing it. So we extract all that metadata, that will be our third step. And the fourth step again, or the fourth pass, is to go look for all these cross references and extract the mutations there so we can capture that and save it into our database. And the fifth step is basically this is one of the key unique things in our project where we are trying to generate new features that have not been identified through these first four steps. So, Linde is annotating these features by looking at one of those samples.

[58:28] But let's say if we get another sample which has a new document type which we have not already counted for, then this step will, using AI, say, okay, I've not seen this document type before. We've got a document, this will put it aside. And then this is where we also think about involving human in the loop. So we kind of like put it into a queue of oh I see these new document types in this folder on page whatever, right? And then archivists or historians or librarians can go back and then approve or reject, oh yeah, no, this is a duplicate of this, mark it into an existing document type or there's a new document type, save it in the system, right? So it's a living sort of self-healing sort of step and which all leads to basically storing it into a knowledge graph, right? where we create entities and relationships and nodes from the data we have collected in terms of classifying it, grouping them, metadata, cross-reference data, and the new features that have been discovered and approved.

[59:30] So all of that form the knowledge graph. So that knowledge graph is now source of truth for anything that can sit on top. It could be a keyword search, it could be a conversational natural language interface, or it could even be offered as an API for other systems to access the data. So that's the idea and we did some brief, like with the sample that Linde was working on, we had created, we had let it go through this fixed-day process and what came out of that is the final step you see here is the Neo4j Knowledge Graph, an example of this, where you can see at the center a name here and that is the person who sent a gift to FDR. And you can see the other nodes on the sides and how they relate to it. document types, right? So each of them point to maybe an account letter or a process note or a filing note. So this gives you an idea but again this is a proof of concept stage so we just used the sample which was already annotated by Linde so

[1:00:31] that we can validate some of this stuff. But we got to put it through all the folders that we have and then see what comes out of it and that's where we're meeting with, you know, Kirsten and Bill from FDR library to kind of validate some of this data to involve them in the testing process basically. But the goal is like if we have the data in this format then it's easier for LLM to look at it and also not just simply do a semantic search or vector embedded sort of a search whereas it's actually going to create a cipher query through prompting and everything. It's going to translate or natural language statements or questions into a Cypher query That will query the knowledge graph and if it does not query, it returns nothing. It's just going to say I don't have information of what you're asking about. But if it finds that, then it's going to present it. And another important thing which we could also augment using this approach is having enough metadata so we can actually provide sources in an explainable manner

[1:01:32] so it can link back to Linde was showing or I think Rich was showing the website that FDR library had created which is public facing. So if we provide links back to that, so the users can actually get references back to the original documents and stuff. Right now they have to go in and manually produce through individual folders to get to that individual document. But if there is a natural language interface, they can ask a question, whatever comes out of it, it will be giving them a link back to the public facing website so they can easily reference. So this is an example again from that particular sample. So here you can see a keyword search based tool right at the top at the bar. It can just say the name of a person, so it brings up that particular record. You can also see here at the bottom it gives a link to the source. So when you click on that source it takes them to the document that the entire PDF that's uploaded by FDR Library online.

[1:02:35] And here is another example where you can actually use it against a natural language interface where you can ask questions about the records that are underlying. So this way, you know, the idea is to enhance access to these digitized collections, but, you know, grounded in the actual data and not just using an LMWriter of the box. So, with one shot prompting, we tried that and there were a lot of misses with that and you could do off these passes in one go, but it's a needle in the haystack problem at that point. So, we had to break it down into these six passes so it could go and reiterate on the individual data that it's looking at with different perspective and points of view and then bring back the data that we need so that we can ground all of this into a knowledge graph. This is the human in the loop step, this is just an expanded or blown overview of the step 5 that I was talking about where you know AI is

[1:03:35] looking at these documents and we have given it a sample of document types and if let's say it finds a different document type or other information that it didn't find in our trained data, it's going to propose okay I found this new set of data or features so what do you want to do with this and then feature queue and then a human would review that and then they would say approve it or reject it or maybe even reclassify it to an existing feature or document type and that gets fed into the knowledge graph and not just that, it also goes and looks at the existing documents which have already been processed to see oh okay, I don't know if I missed this because it's a new document type which I'm not seeing so I'm going back and looking at the existing data and bring that into the knowledge graph. And then it kind of like keeps going the loop. Here is an example, again this is all proof of concept. So this is what the human, the archivist would look in the queue, the feature queue

[1:04:37] that has been discovered by the AI process. And they can say, it also has references to what page and what folder and what file. so they can actually go in and look at the document of oh yeah this is something new, approve it or no this is not new, you know mix it with the other types that we have. Yeah that's pretty much it so we are looking into you know collaborating with this more, we need a lot of community members and we're starting with FDR to kind of test and validate this process so yeah we are happy to work with anybody who is interested. Thank you. Thank you. Questions? Yeah, so we're going a little bit over time and we're going to hang out and... I have one. So I've actually done a fair amount of work in the SDRP. Right? And quite a bit that also relates to what you're doing when it's here, the National Archive, with a variety of cross-references and all.

[1:05:41] So I'm very intrigued by this because I can see as this grows, you can almost scale it up on this basis. But I do have a couple of questions about some things in this whole process. Two in particular. So I'm wondering how provenance is treated. Now, the two examples that you presented are the personal files. It's pretty discreet set, even though it's less discreet in some ways than the more about diaries, right? But how is provenance tracked in that sense or could potentially try to cause multiple records? Is that provenance often forms an essential part of the content for an individual document, of course. And the other thing I'm wondering is the handwriting and the handwriters. Sometimes we can know who the handwriters are. Is that anything that you give it to you to try to track or how you might do it? I mean, some of these are faceless clerks, but FDR had a whole heavy of personal stereotypes. And so we have a pretty good sense in all the time of what their

[1:06:41] handwriting is. So I'm wondering how that might have played or that is a component of these analysis for this specific project. The filing notes, the cross-references, actually often have the typists. The initials. The initials. Yeah. The type there. And so I think that we would train the agent to recognize that and file it away. There's huge documentation, I gather, at the FDR library that document a lot of these process. Bill Shipman wrote basically a whole description of their process. So that has to feed into it. The handwriting is a problem, but I don't think it's an insuperable problem. The handwriting character recognition is improving. that the general profiles could be matched with samples. Some of that is happening to, I'm not the technical person.

[1:07:41] Sure. I'm the overthink what you're looking at. Exactly. So the reason we're looking at the buzzword, but this genetic architecture is that for hand, so there are all kinds of AI models. So part of the process is to figure out which AI models are appropriate with certain document types, including the handwriting part. So you don't want, if you're using Gemini, Gemini to do everything, with an agentic kind of staged, multi-pass AI, controlled sort of AI, targeted AI agents, you could presumably figure out which is the best or most appropriate model and delegate to a particular handwriting recognition. It kicks it out. It kicks it out. So it's this notion of hunting here and a little bit of that. That's where speciality is. We've noticed that there are some really amazing advances in handwriting

[1:08:46] recognition in models that are really precise for all these handwritten notes. So part of the strategy is to figure out what and when and how to kick those into place and how to coordinate them. I think that's where we're going to get some interesting things. So the idea would be since this month passed, we would have an office greater agent within each pass. So it's kind of like thinking of it as nested agents. So we have And then the second pass is where we identify what And in the second pass, it's trying to put it together. It's also going to look at handwritten document and then say, like Richard's point exactly, we would just sort of call a sub-agent which is dedicated for handwriting. So I think one of the Qwen models is actually doing pretty well for handwriting context. So we would

[1:09:47] and then also store some of those profiles, right? So if it looks like this, just have a metadata of that, okay, we're already in 500 with the initials to kind of like tag them together. And then if it sees another document with similar profile and similar initials, then it will be like, yes, you know, map it together as this is the secretary that was kind of handwriting both of this together, right? I mean, one of the interesting things about your fifth pass, right, is I think we're going to, and stenography notes in the on some of those documents. And so in that fifth pass, maybe a time for the agent to also say, there's handwriting here that I can't figure out. Yes, so I think that's the initial idea would be to just, you know, like whatever it can see and identify right off the bat. But if it has not seen the data before, then that's where the fifth pass gets a bit. It's gonna just put it in a queue of like, okay, I found this new thing, I don't know what it is.

[1:10:47] and then tell me what it is so I can put it into the mix. Yeah, I'm struck by it because I know I'm strict for a comment, is that as I was going through it, of course, one of the fears of AI is going to replace people, but what I saw in this, it's going to generate more people. Yes. They're creating more people because there's still the issue of judgment and discernment that yet these technologies don't. So we have been really meticulous in not having AI make any decisions in the whole process, right? So it's just looking at stuff, it's presenting the data to humans to review. Even at the final step, actually, a human is interrogating it. So it's not deciding anything. It's not like, okay, I'm not taking decisions on behalf of some other human. I'm just making it available for you. I think what's emerging, what we're seeing through all these other historic projects, like we have a project with World War II incarceration records, including census enuration forms. and the student was defending her master's thesis in a few weeks, looking at

[1:11:51] census data from Sacramento, California, 1940-1950, extracting Japanese-American ethnicity and names, connecting that with camp records themselves that have not been released to the public. So the point is that you can then track the trajectory of individuals across all these different collections and link them through what we're calling now knowledge graphs, which are a big enabler of kind of deficient AI processes which have no context and don't know about provenance and really can't go that far. But if you can capture, you know, those essential properties and transmit that to even a generic large language model and tell it, and it's not perfect technology, but it works much better. So if you have the so-called RAG technology, which stands for retrieval augmented generation, you're essentially customizing the LLM

[1:12:55] instead of recreating an LLM that understands your historical contents, which some people are doing, but would be very onerous. So you're leveraging existing LLMs by telling it to only... that its knowledge base is that knowledge graph which captures a lot of those records, archival, historical, cultural annotations, meanings, which he doesn't know about. So if you can structure that appropriately. So long story short, I think what we're seeing is that, sure, we could consider using AI and we can't help things out. But to leverage AI, you need all this cultural historic records. That's where you can properly see if you don't have that kind of training. That's pretty obvious. We'd love to, any of you who are interested

[1:13:56] in this. Everyone, if you head to lunch, your table is on the back of your table. Thank you, sir. Thank you, guys.