Abstract

Commercial AI platforms follow a predictable decay cycle, so a methods paper can rest on a model that no one can access. S. Wright Kennedy argues for building with AI rather than outsourcing to it, using locally run tools on data you control, with outputs that leave an auditable trail. His worked example is a desktop text mining and mapping tool built on Melissa Dell's reprocessed optical character recognition (OCR) of Chronicling America newspapers, demonstrated on coverage of the Haymarket affair. The talk asks what provenance means when the tools used to analyze the evidence no longer exist.

Chapters

Transcript

AI-generated transcript. It was produced from the recording by speech recognition and cleaned up with AI. It has not been edited by the speaker and may contain errors. The captions on YouTube are AI-generated in the same way. Each timestamp links to that moment in the video.

[0:05] Thank you all for joining. I'm Wright Kennedy, a historian at the University of South Carolina. I want to thank Louis and the AHA for making this conference official as well. And anyone joining via Zoom, I think this is the camera. Welcome. The title of my paper is Searching for Reliability, How We Can Use AI to Build Persistent Tools, how we can do scholarship that has provenance with this quickly changing landscape. You know, the tools that we use to analyze the evidence no longer exist. You know, we'll use ChatGPT 3.5, we'll use 4.0, and those get sunsetted not on our timeline, but on a company's quarterly timeline. So my answer to this is in one sentence. The most consequential decision a historian can make about AI isn't which model to use, it's whose tool you're agreeing to live inside. And I'll argue today that you should make your own tools and live inside your own

[1:06] self-sufficient tools that can survive for much longer, that are reproducible. When you run the same analysis in two years from now, all the functions of your tool work rather than someone else's product roadmap. Historians, that isn't just a methods problem. It's a provenance problem. So I'll go through four sections today. I'll start with the diagnosis. Why this moment for AI and using AI in our scholarship is pretty unstable. My stance is what I think we should do about it, which is build with AI rather than outsource to it. So if you're uploading any data sets to AI, that method that's being done in the background is, you know, much of a black box. There are ways around this, of course. The toolkit. So I'll give one worked example of something that I've built, some software that's custom built.

[2:07] The importance here is building tools that are built by historians for historians that are specific to the sources we use, the questions that we ask, rather than taking commercial tools that were built for local government, military, whomever else, resource management, and trying to use those to answer some of the questions that we can use those to answer, such as GIS and Esri. So I think this talk in part pairs with the plenary this morning, the discipline we need to build, thinking about what kind of training, what kind of methods we need to take in this brave new world. I want to set up the session earlier, you know, talks about this structural question. And I want to answer one answer. I want to give one answer at the scale of an individual historian's practice, my own. In part, I hope this leaves you with more

[3:09] questions than answers. Let me see if I can get the mouse over here. There we go. And I think the sessions this afternoon, Louis's closing statement at three, I hope some of the questions that I leave this presentation with will be sharpened, complicated, pushed back upon. So thinking about why this moment is unstable, why it's a special moment, how it changes the way that we need to approach this. For those of you that have read Cory Doctorow, enshittification was a pretty good diagnosis, but an even better title, right? The enshittification of the Internet, the enshittification cycle. This cycle is phase one, platforms great to users. It's building a user base. Phase two, the surplus shifts the businesses, the

[4:10] advertisers and sellers. Users become the product in phase two. Phase three, the platform squeezes businesses too. This is maximum rent extraction. And then phase four is collapse or adaptation, having to kind of go back to earlier steps in this cycle. What drives it is lock-in, no interoperability, a weak antitrust. are at work on large language models right now on a compressed timeline. Why does this matter to historians specifically? Because our reliance on these platforms is going to be judged by future historians, by current historians, using whatever evidentiary standards exist now, but also which exist in 2040. If we publish a methods piece in 2025 that rests on a deprecated model, a model that's deprecated two years later or a year later, we're asking those future historians and contemporary historians to take our word for it. I still come back to the Jonas Salk quote, our greatest responsibility is to

[5:16] be good ancestors. Good ancestors, you know, health and environment and all of this, but good ancestors, the future historians as well. So an example of this, are a few examples. We've seen this over and over and over in recent years. Uber lost about $2 a ride from 2016 to 2018 to subsidized fares. Remember, it was the give $10, get $10, sorry, give $20, get $20 off your ride when your friend uses it, and then dropped down. And now, as we're moving into a major rent extraction, Amazon sold below cost until Borders and Toys Spotify had a free tier and absorbed the gap. The pattern subsidized with investor capital, lock in the market, then raise the prices. LLMs are deep into phase one and starting phase two. We see free tiers start to disappear, advertisements being added to the free tiers. The API pricing is still well below the true cost of

[6:19] compute, as has become the term. um rate limits and model downgrades are happening on free tiers and the enterprise tiers are multiplying our university signed a kind of um uh gateway contract you know for a million dollars which is well below the cost of 30 000 ChatGPT accounts um and uh so so we're still in the get the enterprise customers hooked on the product, get them using this product at scale in critical workflows. We've seen data retention terms tightening through the terms of service updates. When we talk about provenance, this is the context. The ground under our methods papers is moving. And I'll talk about a methods paper that I put together a couple of years ago that just went up in smoke because ChatGPT 3.5 was no longer accessible. What's the point of publishing a reproducibility paper on ChatGPT 3.5 when no one

[7:23] can use that model? No one can check that model to see if it actually is reproducible. So three things hold this talk together. First, the model you publish a methods paper about today might not exist in 18 months, as I just mentioned. And second, reproducibility is now a function of someone else's product roadmap. We've spent decades arguing for open methods, some replication, transparency, and the whole discipline's history is based on this idea of evidence, you know, and source and understanding, you know, as was put in the plenary this morning, that evidentiary standard. None of that survives. if the analytical engine is a black box that gets rewritten every quarter. Third, the training data, the inference costs, and the pricing are all moving while you write. The tool you describe in your introduction to a methods paper may have changed by the time you get through peer review. Put those together and you get the version that's specific to our discipline.

[8:23] Provenance ends at the API. And for historians, the gold standard isn't just someone can run the analysis. It's a future historian can trace the chain of evidence back to the source. The tool in the middle no longer exists. That chain is broken. And that's the case for caring about the shape of our tools and not just how well they perform. So a few very cherry-picked examples of some DH projects that died, that didn't survive. And then one project, and I'll mention a couple others, but one project that I think is a model for these types of projects that have this long-term sustainability. So things that died, constellate, JSTOR sunset at this. In 2025, PaperMiner, you know, we were looking at text mining tools for a project that we were working on. And PaperMiner was this mapping historical newspapers. They have all the methods papers published, but the tool doesn't work. And so you can't use the tool and you can't, you know, any research based on the

[9:28] tool, you can't go back to the tool. And then I saw a presentation a couple of weeks ago where someone said, our story maps are all gone. We can't get the data. It's gone. And Esri gave ample warning. You know, if your email is up to date, they gave years and years. If we're sunsetting the old story maps and now we're going to have ArcGIS, not to be confused with Esri's story maps, ArcGIS story maps, no space between the story maps. that they just finished this retirement in the first quarter of 2026. And, you know, a lot of, you know, good historical mapping, historical GIS work had been built in the old Esri story maps, and the porting process wasn't easy. You know, it was very clunky. Most of it was, you know, build it by hand or download, you know, HTML that you can try to get up and running. It doesn't work very well. So one project that has survived for a while, and this is an analytical

[10:31] open source project, GeoDa, for different emphases, emphasis. This is Luc Anselin's project. It's a geodata visualization software that he developed and has been developing since 2003. It's 23 years and counting. 450,000 unique downloads. If you've worked with geographers, GIS people, they should know GeoDa. You can use it as a shibboleth to see what kind of chops they have. But when you install it, you have it. There's no account, no server, no exposure to any kind of vendor pricing. This long, long running scholarly tool has received funding, but every version that they built would still run right so if they didn't get funding to add new features that was fine you still had the the old uh version of GeoDa um and is uh remarkably stable software you know it uh runs

[11:36] on um on mac on on mac os on windows on linux um so so there are other softwares out there um other dh projects that exist transatlantic slave trade database um is one of them that's going strong. I think one of the big challenges, though, are the cloud-based projects. If you're doing a cloud-based archive project, it's not the last version is going to work if you run out of funding, if you don't get the funding. You don't have server space. You don't have the money it takes. In the talk, I heard Vincent Brown talk about the Transatlantic Slave Trade Database. one of the maintenance contracts for that was $700,000 in 2007. So we're talking big, big money to maintain these. Some of you may be familiar, hopefully many of you are familiar with University of Richmond's Digital Scholarship Lab and their American Panorama. To build out three of the maps and talking with Rob Nelson about this, who's the

[12:38] director of that lab, it was a $500,000 contract. So here are the three maps and maintenance for a couple of years, but not indefinite maintenance. In conversations with Rob about this as well, he had said, oh, I tried porting using LLMs to port one of our old projects that was defunct. And it worked. I spent a weekend porting it and it came back a lot. And so I think one of the big changes, It's not necessarily like 500,000. When you have big, when you have kind of top level cartography, top level design, that's still going to cost money. So it's not 500,000 versus a long weekend to build something from scratch. But the maintenance contracts, the sustainability of digital history projects, especially projects that don't have large server loads, has dropped probably the most significantly in the past four

[13:39] years than it has before. Another example of this, I don't know if any of you came across Paper Machines that Jo Guldi and a collaborator built. I don't see Joe in here. in 2012. It was an add-in for Zotero. It built in some of the voyant functions of text mining, of some of the mapping. And they announced on GitHub that it would be unmaintained. I think that announcement went up nine years ago, but the last commits had been 14 years ago. And since then, Zotero is a citation management software, for those of you that don't know, that's one of these great projects out of George Mason. And so that was on version four of Zotero. Zotero, in this past year, since January of 2026, has released three major updates. So they went from Zotero, or four major updates, Zotero 7, 8, 9, 10. We're on Zotero major update 10.

[14:41] And so without some type of maintenance, the old plugins won't work on Zotero 10. As just an experiment two days ago when I was waiting for my delayed flight that was an hour and a half flight and a three and a half hour delay, I said, well, can we update Paper Machines using Claude Code and get it to run? And in an hour, it updated, it ported it to Zotero 10 and it was functional in Zotero 10. I think one of the exciting things about this, and after doing that, I was about to change my whole presentation to talk about more sustainability. But what I want to do in a future digital history class that I teach is have students all go find a defunct project and see if they can port them, see if they can get them revived the projects. And then talk about what parts of the projects are revived, what parts don't

[15:42] revive. So in Zotero and Paper Machines for Zotero, it was any of the most of the database connections, connections to JSTOR and one of the D-based connections didn't work. But but all of the other analysis worked. And so anything based on kind of that cloud infrastructure, again, is really hard 10 years, 14 years down the line, the chances of all of that working. But, you know, those tools that are hard coded in there still work. And so thinking about, you know, when we build new tools, when we do this kind of research, what what we can build that we know will work. And in working in Claude Code and others, you can say, OK, I want to make sure I partition off these parts that are more likely to break, right, that are using APIs from other services that are reliant on something somewhere else for them to work. And so we can kind of compartmentalize those things and build in greater

[16:42] sustainability. So back to this talk that I'll finish writing that talk soon. um so uh again build with ai don't outsource to it um so two ways to use ai in your research uh and there is a third way that was mentioned by uh song yesterday Song Chen yesterday in we can have ai spit out instructions and then we can run it uh in the software follow the instructions uh in the software uh user clicking using a graphical user interface um so so that one kind of sits in between these two. But the first, we can upload an archive, upload a data set into LLM and ask it for an answer, right? You can copy the Python code out, but it's running it all in the cloud. It's, you know, really a black box. It's an easy way to get into trouble with your data, with your analysis. And when you come back to that, you know, a few months later, to try to figure out

[17:42] what it actually did. And I did this when I first got to LLMs and started using LLMs. I said, oh, well, let me throw a data set in there and see what it can do. And then a few months later, I said, oh, man, that result from two months ago was so cool. How did I do that? And, you know, 3.5 was defunct. All of this was gone. And so it was really hard to kind of recreate or to do a forensic dissection of what it actually did to get that result. so that's you know upload the other side again what i'm arguing this talk it can help write tools that run locally on data that we control you're in the code you're in the data you can rerun it in two years you can rerun it uh in 10 years and 20 years the way python's going python seems to be sick sticking around uh thankfully so um and the outputs uh from that can flow into your other tools. And I'll talk much more about these outputs. I think these off ramps in not only our work with LLMs, but in our own tools that

[18:42] we're building are critical for having these kind of breadcrumbs that we can use as an audit trail that we can use to make sure the tool is doing what we want it to, what we expect it to do, what we understand it to do. So as a quick AI companies, OpenAI, Claude and others, they're looking for these projects they can hold up. They're looking for ways that people are using their tools for publicists, you know, so they can publicize it so they can get more people using it. Here's one that came out this past spring, designing delightful front ends with GPT 5.4. And it said, yeah, you can do all this, all you have to do. So you drop in this prompt. Here's the starter prompt. You know, some of the things from this prompt are, you know, for React code, prefer modern patterns, including use effects, events, start transition, hero budget, only

[19:42] the brand, one headline, one short sentence, cards, default, no cards. So all of this, right, it's a bunch of jargon. It is what the author already knows that they're bringing to it. It's domain expertise. The lesson for historians runs in a similar way. Experiment with these LLMs in the domains where you have expertise, where you know the answer already. So that's one of those steps in saying, OK, you know, am I going to get what I expect to get out of this when I put it in? And if I don't, then, you know, either my worldview is completely messed up and I didn't know what I thought I knew, or it's taking shortcuts, it's trying to cheat, it's not understanding how I'm prompting it. So you're able to tell when the model is wrong. Most of us aren't software developers. We're not going to write a delightful front -end prompt, but we can build off-ramps too.

[20:42] So these off-ramps are ways that we can get, there's the big one, build off-ramps. So we have the data sets along the way, right? It's not one and done. Here's the work. Just without the output. No, instead, along the way, you're getting CSVs. You're getting a log of all the operations that are happening, all the settings that are enabled, the parameters for this. And I hope this will make more sense if it doesn't make sense now as we go through the next section. And so this is a piece a software that I didn't set out to develop. I am working on a project with some colleagues at University of Manchester. We found historical weather data, daily data for cities across the U.S. from 1900, reliably from the mid -1890s to present. And we wanted to compare the measured weather, right, the metric of weather to the

[21:43] culture of weather in newspapers using the of Congress chronicling America. So how cities that are having similar measured weather, are they talking about weather in the same ways? So we're working on this project. Where we started? We started with US News Map, which was on my defunct projects list until this morning when I was like, let me make sure it's still defunct because it didn't load for the past year and a half. And no, they have a 2.0 version. I didn't see any explanation about we have a new 2.0. So maybe they're waiting and I'm running their surprise. But but it uses Chronicling America. It's a map interface where you can search for a term and it shows you on the map newspapers that had that term within that date range. So that's where we started. And I had, you know, undergrad class assignments where they compare the sinking of the Lusitania, the coverage in Western newspapers to Eastern newspapers and split up the class. And then they'd say, oh, well, you know, people in New York are mentioning specific names of passengers. And it's a good assignment, too, because the Lusitania is a

[22:48] pretty unique term. Right. So we're able to make sure there's less ambiguity in the newspaper mentions. So the site disappeared, tried spinning it up locally. Library of Congress's API recently was was migrated, was updated. So that may have been when the people at University of Georgia decided to upgrade U.S. NewsMap. The other thing with U.S. NewsMap is you can get a summary of the newspapers and account, but you don't get any other information with that. So it's hard to get out tables or anything else that you can actually run spatial analysis on. Again, Library of Congress, the direct API, we were having so much trouble getting it to work. They were migrating it to the new system, and now it works like a charm. But I'll let you all find that out. Let me know if it's better now. Constellate was a JSTOR product where you could

[23:49] download filtered newspaper data sets. And JSTOR, again, Sunset at Constellate, Voyant, the type of text mining code kits we wanted to use, where the data sets were on the scale of billions of words. And so it wasn't workable with Voyant. And then we used some GeoJSON code that is limited in interactivity. One of our other collaborators, Luca Schultz, is at the University of Manchester. And he's well in the historical GIS and DH world. And then our other collaborator is a senior cultural history of weather and meteorology, Vlad Yankovic. And so when I said, OK, I'll give it a go and see what I can do from the data sets posted to Hugging Face that have gone through. Melissa Dell at Harvard has been reprocessing Library of Congress historical newspapers for better OCR using machine learning techniques.

[24:52] I said, yeah, I'll see if I can spin something up and make a tool. And Luca and I met back and forth, and he did some of the work on the measured weather. And then we realized that us just telling Vlad about, you know, okay, well, we found this stuff. It wasn't going to cut it. And so that's where it kind of morphed into, okay, well, let me see if I can build a tool that Vlad can use, that he can use for the big exploratory process to ask these questions of the data sets. And then Luca and I can do the spatial analysis and do the other parts of this project. And so that became, you know, the goal, the objective of building this data set. And so it has different functions. It searches the data set. Again, this is a very large data set converted into Parquet format as a database. It's a very efficient way to search through structured text.

[25:58] Then we clean the results, the duplicate, analyze the text, visualize and output. So again, this kind of process of developing software, thinking about what you want it to do, what are the key functions? So you start kind of big picture and then you break it down and you start breaking it down into these smaller and smaller chunks. Gave gave shape to this. You know, we want to do we want to do some kind of analysis. We want to be able to work with these records. So then it kind of snowballed into these larger parts of it. You know, we're certainly open to adding more in the future and refining this. The worked example I'm going to give, we're still working on the weather spatial analysis part. So today I'll just talk about the Haymarket Affair of May 1886. which I think is one of these events that was covered very differently regionally. So it gives us a really interesting lens into labor in the late 19th century. And this is the high watermark of the Knights of Labor

[27:01] and sets up the next labor organizations in the early 1900s. So first, the data that we use to, again, chronicling America. images. If you haven't been to Chronicling America, please, please go check it out. It's an amazing resource. They have plain text, but then we went to Hugging Face and found Melissa Dell's work in using machine learning again to reprocess the OCR. After developing this tool and doing all of the data sets and all of that, getting the database ready, I heard someone said, oh, Library of Congress is redoing all their OCR. And I'm like, I don't have to redo all this stuff. But they're using Melissa Dell's work. So that's coming in and said, oh, OK, great. We picked the right one, picked the right horse. So one honest caveat about this is 100 gigabytes of data, which is difficult to distribute. But we did a wide, wide sweep of Library of Congress going from

[28:07] 1850 to the early 1920s. In working with people from my library at University of South Carolina, I learned so much more. They're part of this historical newspapers project. And they said, oh, yeah, most of the funding came out in 2022. And there was a hundred year lag on newspapers and copyright. And so that's why after 1922, you see this significant drop off in the newspapers that are digitized. And there are some that have continued. University of South Carolina has been digitizing some South Carolina newspapers up through 1926. But geographically, you see it fall off. And before 1850, you have a major geographic fall off, too. And so part of this was we want to make sure that the maps that our senior colleague is looking at, he's not going to say, oh, look at all these mentions over here and none over here, when we just don't have geographic representation of those sources on the other side.

[29:09] So here's how the tool opens up, runs on Mac and on Windows and Linux. There's no browser, no account. This is a local tool. I have a hard drive that I have my 100 gigabytes on. But again, if you're working in the 1890s, you can have it on your computer. I just have so much other stuff that it didn't all fit. And so pull up the data set. The Haymarket Affair, as it was called, Haymarket Bombing, Haymarket Uprising, came just after the May 1st labor parade. 350,000 workers across the country went on strike for an eight-hour day. Chicago had a vibrant labor movement scene and brought together radical socialists, anarchists, immigrant workers, native-born, anti-monopolists, this huge conglomerate of labor activists. The strike at the McCormick plant turned violent on May 3rd.

[30:10] police killed four strikers and labor supporters organized the rally for the next evening in Haymarket Square. So we can see this sharp increase in the number of mentions of Haymarket in 1886 related to this event. Mayor Carter Harrison attended as an observer and declared the gathering peaceful, but even so police arrived and ordered the crowd to disperse. In the middle of that confrontation, a bomb exploded, killing several officers, and to this day no one knows who threw it. and more police and bystanders were killed and wounded. The police in Chicago and elsewhere used the bombing as an opportunity to arrest and suppress labor radicals, as we'll see in one of the examples. Some labor leaders were executed, four were executed, and it inspired other labor advocates. Lucy Parsons's partner is one of those, Albert Parsons, who was executed. And she went on to have a long and fiery oratory career for the labor movement. In 1886, this, again, was the high watermark.

[31:12] So we've done, we typed in Haymarket. Again, it's, you know, somewhat unique term. So the search runs quicker. Some of these searches, what's great about Python and this Python-based app is, you know, it's not going to time out. You can leave it overnight. You can you can pull in the million newspaper issues. It'll fill up your hard drive. So I've built in tools to the software where after you're done, you can say, save the data or clear all the data that I just generated because I don't need an extra 20 gigabytes on my hard drive. But it creates a log. So it went through, searched each year, talked about the returns. And you can see, I'll talk more about this, but these are links to the GeoJSON that was saved, to the JSON extract of the text coming from that OCR data. So again, here's that uptick. This isn't normalized, so it's just the raw articles per year. So then that data set we just extracted, we can create a

[32:17] basic map out of it, a heat map. This will give us just a kind of quick overview of where those articles, again, unnormalized, not normalized. That means we're not dividing by the total number of newspapers, but it shows the extent of coverage, right? How that coverage changes. We're on a time slider right now. This is 1888. We can click any of the spots in the heat map. It pulls up a list of the articles found and the cities that fall within that heat map. And then we can, as we'll see in a minute, we can link out to the PDF so we can do a close reading when we find these more interesting patterns. And again, a lot of this was was our senior colleague Vlad saying, well, I want to see the newspaper. I said, well, you can look up the newspaper by the identifier and Library of Congress website. And that that friction, you know, that pain point, not looking at and I wasn't looking at many of the newspapers that pulled up, you know, stuck to the pattern. So it was, you know, how can we reduce friction for researchers, for

[33:21] my own research, in being able to do kind of a good job, right? And so, you know, again, in scoping the data set that I brought in from Library of Congress, from Hugging Face, it was, how do we make sure that we're not, it's not a specious data set? You know, we're not, We're not setting ourselves up kind of for failure for a bunch of kind of misrecognized patterns. So now I move to the text mining portion of it. Again, you know, UI design. You can pay a lot of money and get a much better UI design, user interface design than this. But this is just kind of bare bones. Let's put as many parameters or put the parameters that we need. This started off with, you know, very basic. We had city, state, start date, end date in the search term. Then we added time, so you could filter by time. Then we added stop words, include page counts, all these other things. The stop words didn't catch all the OCR artifacts and these little fragments.

[34:25] So we added the ability to drop some of those words. So here are the cities, right? All the cities that newspapers carried had the term Haymarket at some point. We can select the state. Again, this was designed so we could then look at, OK, we see these two cities have similar weather. We want to be able to filter by these two cities and look at the newspapers talking about the weather that day. But I want to use the Haymarket example as well to say this goes way beyond that. I've kind of gotten in the habit of at conferences when someone's talking about a topic. Type in that topic name and see what the newspapers are saying, see what the co-locuits are saying for that. So these we can see the co-locuits are pulled up. We'll see a table pop up with those. These are all the terms found within a specified distance of that term Haymarket.

[35:26] Right. So Chicago Square riot bomb meeting. And this is the type of thing that you can get out of the U.S. news map right now. We can change the search term so we can say within all the articles that have Haymarket. what are the co-locates for the word Chicago? So what's found close to the word Chicago? And so we can keep filtering down, right, to really target the articles and the larger patterns that we're interested in. Here are all the, I think I went through, but here are all those dropped terms. We can also group terms and all these other things that we kind of, you know, as we went along, we said, okay, well, it'd be nice to be able to group terms. So when you see police and policemen, And those seem like they're the same category. So let's group those for the sake of this analysis. You don't always want to do that, right? And here's where you set the context window. So a context window of five means five words before or after the search term. So Haymarket in this case. We can do other things like bump charts.

[36:26] Ryan Heuser's work on the semantic revolution really got me excited about bump charts in recent years. These show the ranking of terms that are found close to Haymarket. And so this is flipped, right? The number one ranked term found close to Haymarket is at the top, and then it goes down in a logarithmic scale because a lot of the ones that are in the top 10 fall out. Let's see. So right here, I've set the home bin to 1886, to January of 1886. And we can see how those terms change over time right before, sorry, 18, yeah, January of 1886. And so before that, we don't have many mentions of these terms that become the top terms very quickly. These bump charts are useful for looking at change over time, right, understanding

[37:28] how those terms found next to Haymarket changed over time. We can set different home bins. There are all kinds of settings that allow us to look at this data in different ways to try to dissect the data more and dissect these patterns that we're starting to see. In this one, you can see London. So the Haymarket theater in London was the top co-located term before the Haymarket uprising in Chicago, the Haymarket bombing in Chicago. And so we can see this this change. And that was one of the things, you know, if we take all articles that said Haymarket, even in after the uprising, we're going to get some of those with London. Right. So then we can filter out things like one. If London is found close to Haymarket, then we're not interested in that article. It gets harder when things are geographically closer than London's Haymarket Theater and Haymarket Square in Chicago. We can change the date. We can change the temporal units so we can look at weeks. We can look at days. Even days can get a little iffy because we're splitting the

[38:30] data so much more. This is week by week over that year 1886. from the bombing. We can see spies has a run to Haymarket Spies. Riot immediately after the bombing becomes the number one term bound close to the term Haymarket and then starts to drop off. We see police. We get more words about trials, investigations coming in. And then, you know, the big thing, so we can bring this into GIS, but getting Vlad, getting others to look at the spatial patterns on an exploratory scale is where we bring in the co-locates. So before we were looking at just the prevalence of hay market, the term, but now we're looking at the prevalence of riot. So we're looking at the number of newspapers where riot was the top term, the hue, the darker the hue of red, or the redder the hue, sorry for the color theorists, means more articles were

[39:32] published in that city that included this collocate term, riot. Again, here's the example of going, this is based on, you know, I don't have all these PDFs on my computer. It goes to the Library of Congress's direct link to the PDF so we can go and read what that article is about. And so a lot of this was just trying to ground truth, like, are the patterns that we're seeing, the patterns that we're finding exploring this, actually what we think they are. Well, let's go to the document. So again, reducing the friction. We have the time slider for the time bins that we can set. All of that pulls over. This is just opening in a browser, right, to simplify. We don't need a whole mapping software. We can open it in. It's an HTML file that opens locally. I think the base map is being pulled from a server, but you'd still have all the spatial data and you can direct it to a different server if that ever does. We then can start to do things like look at the top term.

[40:33] So we said, well, you know, we can select different terms, but we want to compare the terms and look for regional patterns before we do the kind of big spatial analysis where we might see these things. So this is pulling out the top term from each newspaper, the top co-located term. And we can see based on, as I'll turn off the label in a minute, but the size of the dot shows the number of articles with that term. And so we can compare areas that are saying bomb or riot versus the top co -located term being police or labor meeting, these other things. So it gives us this view where it's not, this isn't spatial analysis, right? This is the exploratory analysis step. we're looking for patterns that we then can investigate with the real spatial analysis in GIS in ArcGIS Pro or QGIS or whatever and so then there's all these topic modeling and topic modeling is out of vogue now people

[41:37] poo-poo topic modeling but I think it still has its uses in checking stuff so it might not be word vectors are working on now. But this let us quickly see, OK, are there articles in these top articles that are talking about something else? You're looking at co-locates it for, I think, thunder and how different cities are talking about thunder. It's either thunder earthquake and using topic modeling, just a basic topic model found, oh, this one's talking about Big Jim Earthquake, who's a Texas politician, either Big Thunder Jim or Big Jim earthquake. And so, you know, able to, to, again, kind of disentangle some of these results. This isn't, this isn't the evidence that we're using to make the arguments. This is, again, this exploratory data analysis that we want our collaborators to be able to use that, that I think, you know, we can build these

[42:40] types of tools that open up a lot of this work to our colleagues that, you know, when they get to Programming Historian, it says open command line, open terminal, open PowerShell, and there's no, I'm out, you know, load mallet, nope, I'm done. But this is, again, ugly, but more familiar interface, you know, the graphical user interface. And so now we can do things like bring the topic across time too. And looking at the timeline of these different topic models and then building out maps from the topic models. Again, all this is, I'm going to keep hammering this on, this is the exploratory part. And this is how we were checking the results that we were getting. What we were seeing in other parts of the software is how we investigate it. Is it doing what we think it's doing?

[43:41] familiar idea with python and it used to be where i'd go in and uh claude code or codex would write something and be able to find the problems you know oh yeah well like the criminal says this is the error and i went and looked at that line and yeah sure enough that's the error and here's where it's wrong now uh you know that was a year and a half ago now uh claude code and i'm sure codex can do all that work on its own. And it really, it puts out stuff that works, you know, four out of five times now, whereas before it worked like two out of five times. I did some of this development work, and I'll show you a few of the tools in a bit. Before, just using the chat window, you know, browser chat window, I paste in the whole code and then get a code out and drop it into VS Code or run the Python script. And that worked one out of eight times or something.

[44:43] But if you stuck with it, you could get it working eventually. So we've come a long way in the past year and a half, all of that's to say, where building out these types of things is much, much easier. So again, same type of maps. just with topic models instead of co-locates. And really, we can add on other results based on the locations where the newspapers are published. We have this inherent spatial locator or discrete spatial location connected with each of these articles. Then we can build one of the important steps of all of this. So, again, it was building tools into this that let us check other parts of the software. off ramps as well. So that's building the CSV. So this is, you know, I'm going through these analyses. These are the maps that were generated, the HTML maps. And so I can go back and open up that map. You know, all the steps along the way, this is where it can balloon out.

[45:46] You know, if you have, you know, 100,000 articles, it starts to add up because the text is stored inside the the JSON and the HTML. We have the parameters that were used when setting this up. So this is a list of the parameters that's included in the documentation. And then we get things like this. So this is that main window again, and we get the project log. And so this shows that initial search. This shows the terms that were dropped, all the runs, again, all of that audit trail so we can go back to any of it. and we can, you know, in building this, it was so critical to be able to figure out where it was breaking, you know, is it, you know, I'd go and search Library of Congress for a term and say, okay, you know, I found a result in Savannah, Georgia, but I don't see any Savannah, Georgia results on my map and then could pull up that search on here and sort through where the issues entered the

[46:49] software. So again, that's the audit trail. And I think that's one of the answers to the provenance question. You know, this lives on my computer. It's on GitHub too. I'm not sure if it's publicly listed yet, but it will be soon. But the software runs on my computer. The only thing that needs the internet? Are those base maps, the background maps? Everything else, all these data sets are on this computer and filling up this hard drive. So that is where I add it. You know, I'm thinking through just like, what does it mean to save a project? Do we just care about the project log? Do we care about these other things? You know, I'm sure software developers have been thinking and user experience people have been thinking about this for a while. want to put in the top menu settings versus what do we want to have as a checkbox on here. What do we want to really warn users about when they're going to change it

[47:50] and it's going to, you know, fill up their hard drive. So I set threshold limits too. So if it's going to be more than, I think it's default set to like two gigabytes, it warns you. Hey, are you sure you want to do this? This is going to be a two gigabyte file size just for the initial extract. The JSON log also means that someone else can trace exactly what I did. It's not glamorous, but it's just, you know, the step, step, step. Here's all of it. So this is what building with AI rather than outsourcing to AI means in practice. The AI helped me write the export code, but the AI never sees the output if I don't want it to. I do this, you know, a lot of this research now in my own software. I'm not reliant on tools that break, tools that disappear. You know, so it's, you know, a lot of debugging and writing parquet. I was corrected a few months ago that not parquet, it's parquet.

[48:53] In New Orleans, they would say parquet. Yeah, the French that comes out of New Orleans. Um, so, uh, so now this is, uh, you know, the AI can, can help do a lot more. Um, but, but the stuff on this side, uh, is going to stay on that side, right? Picking the keywords, selecting the corpus, choosing appropriate analytical tools, um, deciding what counts as meaningful co-location. You know, I was digging through the Voyant documentation. Okay. Which, which of these Voyant tools, um, you know, are going to help us, uh, dissect the How do we bring this in? And then, you know, chat, chat with Claude about it and say, you know, how are there any Python packages that already have this that we can bring in? The other part of this is, you know, as we went through and this is not the least efficient, I can think of more inefficient ways, but it's very

[49:53] inefficient way. just a OneNote document. And I found that if you use checkboxes on OneNote, it maxes out at like 50. It gets really buggy, at least on Mac and stops working. But this is what it was. You know, go through and just create a list of stuff. And then I'd parse out that list. It was, you know, how do we filter this way? Here are some of the bugs that we found. And just this iterative process of going back and forth to build in new functionality and then test that new functionality and make sure that it worked. So, again, I talked a lot about, you know, when the AI was wrong, when it screwed up the code, you know, the best sign that it screwed up the code was this offer didn't open, you know? And so having good backups using GitHub, using the versioning in GitHub really makes a difference. And if I'm speaking Greek to you, if you don't know what GitHub is, you can go to Claude and say, yeah, I want to use GitHub. How do I use GitHub to do versioning?

[50:54] How do I want to make a change? I want to update this piece of software, but I don't want it to corrupt the main thing. So how do I make a backup? And it'll tell you, oh, GitHub's great idea. You know, that was a great idea to use GitHub. But, you know, so much of this is kind of learning parts of software development by doing it. learning, you know, not the kind of on the ground coding as much, but the questions, some of the practices that save you when things go off the rail. And then a lot of it's breaking it. You know, you break, it doesn't work. You figure out why it doesn't work. The tool works, but it doesn't show the data you expect. So you figure out why it's not showing that data you expect. And so doing these spot checks on random outputs on the first run every time that we updated the tool. So some of those were, you know, you get kind of a group of

[51:56] events that you could go back to and find and check those and then add in a couple extras each time, a couple of random ones. So we have a few more minutes. So I'm going to quickly just we'll breeze through this in a couple minutes to leave time for a couple of questions. So these are some of the other tools that I've been building. And this goes back to about 2023. Text split. We found that early on trying to use GPT to parse text, it would start hallucinating after about 100 lines. split it at that. It was the context window. Then it fixed most of the hallucinations in 2023. But again, like, you know, if I publish this, it doesn't matter that much for 3.5. It matters for 3.5, but no one has access to 3.5 and all that.

[52:56] So this was the article that we were working on and built this text splitter tool to help structure women's social club directories from New York City in 1920. Alexandra Miller really took that project on and did most of the history, most of the mapping, and has presented some great papers on it. And so this just let you set a token limit upload a text file that we were getting from HathiTrust. They're from archive.org, and it would split it out into a bunch of other text files. And this was browser-based. It is browser-based. It still works in JavaScript. And what that means, it runs on your machine. It's not in, your data is not coming to me, right? So you put it in here, it stays on your machine, you get the output, you save it, and then it's gone when you close the window, and it's just saved on your machine. And so this, again, allowed us to get around a lot of the hallucinations earlier on. Other stuff, when I teach one day GIS workshops to historians, the tabular join is where everyone gets stuck.

[53:59] You know, we always get the NHGIS, National Historical GIS census data from IPUMS, and you get the geometries, and you get the table with the data, the CSV, and you need to join it up. And that's where everyone gets stuck. And so if you want to use Arc online, you need a paid account. You can do it in QGIS, but having people install QGIS for this one step where we can do everything else browser-based is another type of challenge. So again, another basic JavaScript web app that does the tabular join. You upload the shapefile. You upload the CSV. These are all zipped as they come from IPUMS. And then you pick the fields to join on. You join the data and you get an output. The one problem with that is that the output's GeoJSON right now, which increases the file size significantly. Other things, TitanicMap, working with the class to build everything online. So this was a couple years ago, too.

[54:59] prompt asking it to build a Python script for the web map. Then we can go to Google Colabs, where you can run Python online. ChatGPT can do most of this, but on its own now. The only thing you had to log into was Google Drive for this to be able to upload a file, but then it generates a leaflet map that's not pretty, but it's an interactive map. And that used to be one of the main barriers to getting, you know, good projects, good historical GIS projects on the web. Students then put this on their blogger account, embedded the iframe, and were able to work with them. And this was really an iterative process. So yeah, great. I have a map of the points where people departed the Titanic. Can I add tables? Can I add a pie chart to the pop-up. Can I do this? ChatGPT says, you know, great idea. Way to go. And it adds it in. So we just change one line and get the HTML out.

[56:02] And actually this one, yeah, there it is. So we get the pie charts and we can make those interactive. So just keeping going back and forth and seeing how far we can push this to do new things. The other one that I'm excited about, And this was, you know, on Jo Guldi's Paper Machines, a Zotero plug-in. And I was saying, you know, that was great to, you know, be able to map all the people that work together, the knowledge networks and all this. I was going to give a talk at University of Minnesota to the IPUMS people. And they're spread across all these different departments. So I couldn't just go to the history department website and say, OK, who do I know? You know, who have I been reading there? Who do I want to meet with? With Zotero, I have everyone I've been reading. So I built a plug-in and I can type in Toronto. I guess I was going to Toronto. I demoed it at Toronto after Minnesota. My story's not holding up. But this does use an OpenALEX

[57:03] online database to look up scholar citations. But what it shows you is the scholars that are in my Zotero library at University of Toronto currently. And so I've been iterating on this as well, just as a tool, you know, going for a visit to a campus. How can I make sure I don't miss anyone, you know, make sure I don't read an article in my Zotero a few months later? But, oh, my God, I was just right up there. You know, I could have could have met with them. and then some other teaching tools building out raspberry pies which uh yeah probably could have done over a couple years but using uh llms uh for that kind of instruction log on you know i have a little soldering and stuff but you know uh and students in my class weren't as big of fans of having to tap their id to take attendance every day um and you can see some of the fall off in attendance over the semester. But thinking about ways that some of the ed tech, so like

[58:05] PeerCeptive for peer reviews, we're charging $25 a student, $24.50. So it comes out to $5,700 a semester, either paid by students or paid by our department. And then, you know, using LLMs to walk me through building a power automate flow that assigns peer reviews randomly to students. And so it's a lot of kind of the tools that have already been there that have been, you know, just a high barrier to entry that we can now kind of get around these barriers of, you know, cost of, you know, software as a service company. there was a big Adobe Creative and Photoshop are there's a provocative piece that said software is dead we can develop all of it on our own a couple days ago so anyway here are the questions that I asked for tool validation does it answer questions you couldn't otherwise ask do you know how to tell when it's giving you the wrong

[59:09] answer can someone else reproduce it reproduce a result with it which parts will break, which will still work? And does the outflow work with your other tools or is all the data trapped inside your tool, inside an LLM is kind of lost or very hard to get to? So thank you for your patience and listening to the talk. Yeah. So we're building tools not for our research, but support for the surgeons. So I'm really curious, I'm a bit of an outsider, what is the life cycle of like, so I say two years, but I'm thinking 10 years, one year. Yeah, that's why I love to, you know, I love to take on the Paper Machines test. You know, that's 14 years, give or take, since last development.

[1:00:10] And it comes back alive, right? And so by running that in my next digital history class that I teach, I hope to have a better idea of that, right? I mean, we already know sustainability-wise that stuff based on APIs, stuff based on database calls, based on super heavy server loads, big data that you're trying to serve out, isn't going to last a while without constant funding, constant maintenance. But something like this, you know, it's Python. I think Python is one of our best bets for the future. So I don't see that going away. And because it's written in Python, I think, you know, future LLMs, if we're all still alive, we'll be able to work with that easier than more niche languages. So, yeah, I mean, it's it's Vincent Brown said that he viewed the slave revolt in Jamaica, this this kind of flagship historical GIS project

[1:01:14] that he built as a performance. You know, and that it lives and then it dies. And so the idea of maintaining it indefinitely, he said, you know, just funding wise, all of this stuff, it wasn't doable. But I think this opens up a lot more options for keeping things alive rather than, you know, a huge maintenance contract. yeah I think you can work with your local reservations department if you have one so you're really sure that you know if you can replicate operating system and stuff work if you had to download but I'm thinking about like if you know our needs of your community and if we build something today uh I know I can see you expect it to be available at two years but it's Is that window too long enough? Yeah, I mean, some of it you don't want to be fully boxed in by what's

[1:02:15] going to last. If there are resources available today, APIs or Library of Congress API that is working and fully migrate it to the new system, then we want to be able to use that for our research now, but knowing that that part of the tool is much more unstable. historical New York, which is a big historical mapping project, we could link in to georeference historical maps to all these different things that were stored elsewhere. But we said, okay, those things break. We've seen web maps, historical GIS web maps, that pieces of them break. You know, if you don't have a base map, it makes it much, much harder to interact with the historical GIS. Yes. But if we if we can bring everything together, then this will be much more stable, much longer. And so I guess I'm not. And for those online, the question is about sustainability and looking at a 10 year

[1:03:15] window rather than a two year window. Yeah. Yeah. Let's say 20 years. Let's aim to make it work in 20 years. Right. But it's which parts of it are going to work. And we have a graveyard of projects to do postmortems on that I haven't done yet, but that we can understand what's going to break, better understand what's going to break and what we can revive. And then, you know, we can use that to design better informed designs of projects, kind of the back end designs. We know this part's going to break, so we don't want to base everything on that part. We're going to have a fallback or we don't know it's going to break, but it's a much higher likelihood of breaking or something changing or a company charging for access to the API, which happened with a lot of the Google Map projects. Did that get at it? Yeah, that's so cool. I mean, you don't often hear good news about project sustainability. So, you know, we take one silver lining of the end of the world.

[1:04:19] Yeah.