Abstract

A conversation on how historians can approach and collaborate with computer scientists: why computer scientists find historical material interesting, why they are often confused by historians' methods and questions, and why historians are interesting to them in turn. Field describes a project with sociologist Steve Morgan on five years of Baltimore police radio, where automatic speech recognition (ASR) fails on police terms. Harrower describes automated detection of archaeological sites in satellite imagery, with Bronze Age tombs of the Arabian Peninsula as the example, and a project with Carnegie Mellon University on archaeological databases. Ippolito explains that AI researchers want verifiable rewards and advises first testing whether Claude already solves the problem. A long audience Q&A follows.

Chapters

Transcript

AI-generated transcript. It was produced from the recording by speech recognition and cleaned up with AI. It has not been edited by the speaker and may contain errors. The captions on YouTube are AI-generated in the same way. Each timestamp links to that moment in the video.

[0:08] Okay, hi everyone. So we'll go ahead and get started. So this session for the next hour is the topic is working with computer scientists. And this is the plan that the three of us for this session came up with. What we thought we would start by is we'd just take about 10 to 12 minutes each to introduce ourselves, share a few thoughts on the topic of the session and hear what I specific my specific motivation here was like I have a chance to communicate with like potential collaborators what I want them to know about my field before we start these conversations and then what we'll do for the second half is just have open Q&A so hopefully we're giving you enough context that you understand like what kinds of questions we're going to be able to answer and not great okay so I'll start so my name is Anjalie I'm an assistant science department here at Hopkins. I've been here for about three years, and what I work on is mostly AI with text and language data, which used to be called natural language

[1:10] processing, and then within that I focus on ethics and computational social science. And just for like a little bit of extra background, most of my training is in computer science. That's what I majored in undergrad and what my PhD is in, but I did also study Latin and ancient Greek in college, and so have like a little bit more of a mixed background. And then I've collaborated a lot more with social scientists than with folks in humanities. So currently have active collaborations with people in sociology and public health, and in the past have worked a lot with people in political science and psychology as well. And of my interest, the ones that I think are most relevant to this event are on large-scale text and speech corpus analysis questions, particularly developing methods to do that. I also work in AI privacy and user-centered evaluation, but I think that's probably not going to be as relevant. And just starting with the kind of prompt for this session, what do I find interesting about historical material? I think it's two main things. So one is it's real data. It's on important real-world topics, and that may sound obvious to a lot of you,

[2:13] but a lot of computer science, we kind of build these fake datasets for the purpose of testing models that don't really reflect real things. And so I think it's really exciting to work with real data. And that also introduces a lot of interesting technical challenges because a lot of times off-the -shelf tools don't work that well on these kinds of real messy datasets. And if tools do work, then there's not a lot for the computer scientists to do. But when they don't work is when there's sort of more opportunity for us to innovate. All right, so what do I want my collaborators to know about computer science? I'm going to try to keep this brief, but one of the, I think, biggest things is that in our field, all of the real work is done by students, and this was kind of a painful lesson for me when I first started a faculty job, and I thought I spent all this time training how to do research, like I'm still going to do some stuff myself, and I quickly realized that if I don't have a student involved the project, it doesn't move. So when I'm talking with potential collaborators, a lot of the

[3:14] times the first question I'll ask is like, well, what do you want? Like, what are you looking for? Are you looking for some understanding of the field, like higher level advice? Or do you want someone who's going to actually help you implement things? And if you need someone to help implement things, then we have to get a student involved. And when we start looking at students, we have kind of two options, right? We can get, look at graduate students or undergrad students. And when we look is true both for master's students and PhD students. One thing that's really important to recognize is that they really have to publish. So in our programs, our master's students, a lot of times they're really great and not all of them get involved in research. The ones who do get involved in research almost certainly do it because they want to apply for PhD programs. And you are not competitive as an applicant for a PhD program in computer science if you have a master's degree and you don't have a first author paper in a high profile venue. So they really have to publish and they have to publish like within a year with, you know, sometimes it doesn't work out

[4:16] and you can write a really nice letter and they can kind of kind of get through anyway. But if we're not starting them on a project that has potential to be publishable, like we're not treating these students properly. And PhD students, like they don't have to have as much time pressure, they have more time. So if, you know, they're getting papers out of this further down the line, that's okay but if they're spending a lot of time on a project that's ultimately not publishable well it's normal for our students to graduate with like five to ten first author papers and so they can't devote a whole year to something that's like not gonna further what they need to graduate so what that means is when we're starting collaborations if we're getting masters and phd students involved we need some component that we can kind of break out into a cs paper and usually like it never is it's never this clean in practice but the ideal is like you start with some difficult data set and current methods don't really work so we come up with some new method and that turns into a cs paper and what's important is for us to be able to show it's like reusable across different data sets and then we can like use

[5:19] that method for the analysis that we want in the paper that's maybe like longer running um i don't think i've ever had a project that actually follow that model but we always think it's going to follow that model. I don't think our papers are really short by all of your standards. Yeah there's less work like you need to do less work for it to be considered publishable. This is a good point and I should say this I'm really thinking about if like assuming a project's not funded like if you have money to hire someone then it's a little different like you can often find a master's student who's like willing to do this for a part-time job. So what do you do if you don't really want to do methods innovation, but you do have something that you just want someone to help implement? Like you want to run all of your data through Claude, and you need someone to help you set up a pipeline for doing that. In that case, I think like a kind of underused resource here are actually undergrad RAs. So at Hopkins, we have really good undergrads, and they can be like a little bit hit or miss that just how much time they actually are going to put into the

[6:23] project, which sometimes you can't tell. But they don't necessarily need to be first author on a high-profile paper. Just research experience at all is good for them. They can often do it for course credit. And so we've had some really great experiences with undergrads in the last year or two. And yeah, that may be what you're looking for. Depends on the project. Great. Okay, so that was my kind of little soapbox. Um, now what I'll say is just a few words about a project that we're currently working on that I'm excited about, um, that's interdisciplinary. And this is a collaboration with, uh, Steve Morgan, who's in the sociology department here at Hopkins. And Steve has this amazing data set. So he basically recorded, um, every police radio communication in the city of Baltimore over about a five-year period. and so it's this like amazing archive it has a lot of potentially really rich information about policing activity in the city he's already has an initial paper

[7:23] showing how by just looking at volume of transmission you can see things like impacts of policy changes in the department and how that actually changed on the ground police activity but it would be really useful if instead of just looking at volume of footage we could look at what these communications are already saying are actually saying. And that turns out to be really hard. If anyone, have you ever listened to radio data, you'll know it's really hard to understand what people are saying. It's hard for people. It's even harder for models. If you start with like a transcription model and you try to transcribe this data, it just does terribly. This data is really different from where a lot of these systems are trained and what they're really actually good at doing. And so we've been working on this data to try to improve what we can understand from the data when we try to process it using models, because we can't hand transcribe five years of data. That's never going to happen. And we have two projects of this. So they're both accepted to Speech Language Technologies, which is a venue for speech processing research. And the first one was led by one of my PhD students.

[8:28] That's the chart on the left here. And what she was working on is broadly, can we just improve the transcription quality? you word error rates compared to straight off the shelf system compared to some tricks we try to do to improve the performance. And the project on the right is drawing from this idea that in this data, we actually probably don't care about transcribing every single utterance correctly. What we want to be able to do is transcribe the data, run some analysis over what we transcribed, and we care about the correctness of that final analysis. don't necessarily need every transcript to be correct. We know our end conclusions to be correct. And so that's what the second paper was looking at is this is looking at if we have an unreliable transcription model that has some bias in how it transcribes data. So this chart is sort of the motivating figure. The green bars reflect data that is frequently, words that are frequently

[9:28] missed by the model. And these are all like common policing terms. Like this is where the model tends to mess up, it messes up on the police lingo. So if we have a model that's not great, can we use some like post hoc bias correction methods to still get estimates of word distributions across the corpus that we actually trust? Okay, so I'll stop there and I will pass it off to Michael to give his introduction. Okay, hello everybody. I'm Michael Harrower. I'm a professor of archaeology and the Director of Undergraduate Studies for Archaeology, and my home department is Near Eastern Studies. So basically, I study the ancient Middle East. And for me, that includes a lot of work in the Arabian Peninsula over the past 25 years or so, and also the Horn of Africa. And so I'm not a computer scientist, but I've been involved in lots of big collaborative projects over the years, including with engineers, computer

[10:29] scientists, statisticians, and sometimes learned the hard way about those kinds of collaborations. And part of this is some of the things Anjalie is mentioning about explaining the motivations, the standards, the practices of your discipline to other very different disciplines. And I've been working with Daphne since, how long has it been, two years or something like that? Yeah. Year and a half on some work that I'll talk about briefly. But so some of my collaborations go back a little ways. And I've done a lot of work with satellite imagery, specifically detecting archaeological sites and satellite imagery. Some of that is based on infrared imagery, but a lot of that is based on just detecting objects. And so this is quite similar to detecting objects in just photographs of things as well.

[11:30] So we did a bunch of work in the late, well, early 2000s that was published in 2013 on sort of automated detection of objects. And I'll show you some examples of that in a second. In 2013, we thought, well, geez, we could push this a lot further, but we're going to need a lot more research funding. We're going to need a lot more collaboration from engineers and statisticians and whatnot. So we thought, well, let's just take a pause on it. When we re-came back to that line of work almost seven or eight years later in the early 2020s, the technology had advanced dramatically. And you now had computer vision detection of objects in satellite imagery. And so that just really dramatically advanced our work. So I've also been the past year and a half or so involved in some work using AI to analyze archaeological data and data in archaeological databases.

[12:32] And specifically, one of the things we've been starting to do now is think about archaeological benchmarks for AI. And one of the things we recognize is that a lot of the LLMs seem to have a lot of shortcomings surrounding the humanities, right? So they're trained on other sorts of issues that don't oftentimes focus in on archaeology or focus in on humanities or, as many of you are discussing, on history. So we've got a lot of dramatic advances in computer vision and LLMs, of course, that are having a big impact in archaeology. and you know part of this too where it really starts to get me interested in is a role of the humanities history in archaeology in fostering constructive dialogue mutual understanding building empathy around shared appreciation of histories

[13:34] I'm an archaeologist that works in the Middle East. This is a really troubling and difficult, tumultuous time. So this is part of my research and writing is focused on the human aspects of appreciating the past. And if we dial back history to millions of years ago, there's some point at the past where we all originate from a common homo sapien ancestor. And some of this really deep history work has a way of, you know, sidestepping the controversies that come up in recent history to think about the deeper human past. So there's lots of intangible benefits here in a sort of politically fractious and rapidly changing worlds to the human aspect of the humanities and of archaeology and of history, of course. To touch on some of the things I want my collaborators to know about the humanities

[14:35] and archaeology, you know, quite normally in the humanities, there's a lot less funding available than in the sciences, and research is often performed without being supported by a grant. I have lots of humanities colleagues over here in Gilman Hall who, you know, with a library card and an office and an internet connection, they're pretty solitary researchers, right? And they do a lot of the work, unlike in computer science, which a lot of the work is performed by students. A lot of times we have far fewer numbers of students, including graduate students. And a lot of the even the basic tasks of working with, for instance, ancient Middle Eastern cuneiform tablets that date back four or five thousand years. a lot of that sort of really hands-on work is done by faculty themselves. We produce a lot fewer publications that tend to be longer, that span many

[15:38] more pages, even books, and have far fewer authors. And, you know, it's bizarre to explain to scientists and engineers, including computer scientists, how, you know, you'll be on hiring committees or grant review committees in the humanities. people will look at publications and think, why are there so many authors? Like, how do I know who did what? And how do we know that this student really did the work themselves, right? So there is a lot of stuff that still gets published by single authors or by two authors. That's really not the model and not the way that research gets conducted in other fields, but it is still fairly common, especially single-authored books, which might take Years and years of research might take five years or even a decade to develop. And then also research is often ideographic rather than nomothetic, meaning that the objective oftentimes is to illustrate

[16:43] the richness, the detail, the historical specificity of one particular case, not necessarily to identify general patterns and rules and generalizability. of the things that historians are doing are conveying a deep insight into a specific moment in history, whereas a lot of times in the sciences, you're looking to develop things that could be generalizable and could help identify bigger patterns. So like I mentioned, a lot of my research over the years is focused in the Arabian Peninsula, in the 1990s and early 2000s in Yemen, in Oman, over the past five years in Saudi Arabia, also the Emirates in different parts. But one of the things I want to point out here, I'm not going to go into all the details of the satellite imagery work, but one of the things that archaeologists

[17:44] have been learning over the past 25 years is the broad patterns of history over the last 5,000 years in the Arabian Peninsula. So a lot of the tombs I'm showing you that date back to about 4,500 years ago show up in the Sinai Peninsula, show up in northwestern Saudi Arabia, show up in Yemen, Oman, the Emirates. and these are what come from a period we call the Bronze Age. And interestingly, at the same time, these were being built and we don't have precise estimates of how many, but you're talking hundreds of thousands of these across the 3 million square kilometers of the Arabian Peninsula. And these are built at roughly the same time as the Egyptians are building the pyramids. So while the Egyptians are building one gigantic burial monument for an individual pharaoh, Arabia, you're seeing, you know,

[18:45] we don't have exactly precise numbers, but in a rough, approximate way, an almost roughly equivalent amount of labor devoted not to one burial monument, but to hundreds of thousands of small burial monuments that are very sort of egalitarian in their ethos, right? So there's, you know, three or five individuals in these. there's women, there's children, there's small numbers of grave goods. There's a lot of standardization in terms of the style, in terms of what's interred in them. And these eventually develop into larger tombs that have hundreds of people interred. We find people, we find men, women, children. We find immigrants that are coming from foreign lands. We find people with So it's almost the flip

[19:46] side of the pyramids for the pharaohs in which only one person gets interred. In these, they're very community-focused in which there doesn't seem to be any part of the population that's excluded. So understanding just the societies and the communities that built these, one of the things that entails is identifying the patterns geographically. aren't they found? We know that there's some connections with water. There's some connections with trade routes. And this is part of the research is sort of identifying where they are. And because only a tiny fraction of these hundreds of thousands of monuments will ever get excavated. So more recently in the past year and a half, we've been working with Carnegie Mellon University and a small group of scholars looking at archaeological databases.

[20:47] And some of the things we've been working on through that are, you know, a lot of these databases, a lot of specialized archaeological data is held in databases that aren't really open to the internet quite so much. There's a lot of sort of messy data that's not standardized from different projects that requires a lot of jargon, a lot of expertise to understand. And we're thinking about ways of and working on ways of extracting insights and identifying patterns that would help archaeologists make best use of this information. But I think I've probably used up enough time here, so I'll pass it off to Daphne, and maybe she'll give you some additional insights. I did my PhD in computer science. I'm an undergrad. I also studied archaeology and geography.

[21:47] My current research interests are relevant to all of you are building, like, so there's lots of interest in, among AI people right now, in building, like, deep research agents, data science agents. These are agents that you can give a research task, and they will do that research task for you. And this kind of works for some research tasks, but probably not the ones that any of you in the room are doing. So you'll see research agents from OpenAI solving a really hard math problem or something that the AI people are excited about is AI making new AI. It's this evolution where your agent can write the codes and build a new AI that's better than itself. Something that all the philanthropic people are both concerned about and also trying to move towards. Sorry, Nabi. I think your mic is muted. Okay, sorry. I'm just putting my back on. All right, so sorry to the audience, the online audience. Yeah, so one of the differences between the fields I

[22:51] just described and what I suspect all of you do is that there are nice, verifiable rewards, which is something that computer scientists really like. They really like when they can build an AI agent that can work towards something where they can measure, like quantitatively, objectively measure, or not. And that is not something that I think your disciplines have nearly as much. And so I think that's really an area of opportunity. That's something that Michael and I have talked about a lot in this archaeology collaboration is what kinds of questions would an archaeologist want to ask of an AI research assistant? And can we build tools that can actually provide satisfying answers for an archaeologist to those questions? So that's one of the reasons why I'm excited about like this sort of intersection is AI research have, researchers have this preferred paradigm of verification oriented research. We really want things where we can verify correctness. You can't verify correctness for sure of your hypotheses about the past, right?

[23:54] You can't go back and like run controlled experiments, like things like that. So that's kind of like exciting and hard. And then something that I also like, and that I think Anjalie mentioned as well, is that AI researchers have this expectation, not all of them, but a lot of them, that all knowledge has been nicely digitized and that all of the knowledge that is worth knowing is already available to Claude. And maybe that's true. I don't believe it's true, but certainly I have literally had students in AI say that to me. And so I think pretty much all of you have access probably to some knowledge that Claude doesn't know about unless you tell it about. But that It's not obvious to some of our students. And then also the fact that data is messy, like handwritten. Like I went to the session this morning on like the FDR archives, which aren't even that old. Like these are like fairly new and they're still like incredibly messy. There's annotations in there, which made sense in the 1940s, but might not make

[24:56] sense now. And this idea that things are not, even if they're digitized, it's not easy to like access and organize the information is something that is like kind of weird. to computer science students because their idea of data is it's either in nice tables, like nicely formatted tables where you can just grab each column at the table, or it's just unstructured web text. Those are two kinds of data that they know how to handle. And actually, you feel free to disagree with me or add any flavor if you have thoughts on that. So I had a different slide here yesterday, and I made this slide this afternoon after lunch because that this came up in conversations at lunch. It's like things that I think are most different between our CS students and history students is someone at my lunch table literally said this phrase. It's like history PhD students are looking to find a niche, like something that they are the expert in and nobody else is the expert in. This is their niche. And if they're lucky, this will be like 30 years of work on

[25:56] this one niche that they discover as a PhD student. And please correct me if I'm wrong. That was my understanding from lunch. that is not something by and large our students who want to work in AI are in they all want to look for research of maximum impact and even if that area has thousands of other people working in it that's okay because it's an area of maximum impact and so they tend this is not all of them but this is certainly at CMU where I'm a professor the modal student wants to work on whatever the top 10 terms in AI Twitter are and whatever the current the current year is and so right now that's Safety is the buzzwords. Agents are the buzzwords. Reasoning is a buzzword. And that's where all the students want to work. And the idea of finding a niche that nobody else works in is completely foreign to them. Because if nobody else works in it, then it can't be interesting. They're not going to have impact. They're not going to get their jobs at Anthropic or OpenAI. And so that's like a very big cultural difference. Another one, which I'm not going to spend too much time on, because Anjalie already

[26:57] talked about it, is CS students want to build general methods and solutions, not problem-specific ones. But looking at all of the work that I've seen people talk about so far today, it seems like historians are much more interested in answering specific research questions. And often answering that research question will involve finding one particular data source, like one archive, one collection of materials, and really doing a deep dive into that. If you're going to use AI at all, you are building an AI tool that is good for that piece of data that you have decided to focus on, which is not something that really gets our students' papers easily, and they all want papers, as Anjalie said. I've already kind of mentioned this, but when I have tried giving history or archaeology-related projects to mostly CS master students at my university, they all ask me questions along these lines of, okay, I'm going to try to build the method that does what you say. How do I verify that it's correct?

[28:00] and very often they can't verify that it's correct, and maybe verification will be like you should go and show the results to a historian or archaeologist whoever you're collaborating with and ask them, does this look good to you? But that's a very slow process. They can't do that when they're programming at 3 a .m. and they're in their apartments. So it's not sufficient for them that let's go talk to an expert. And so we do need to come up with ways to capture what you all think are good and somehow turn them into things that our students can measure in their apartments at 3 a.m. when they don't have a historian or an archaeologist to talk to. um and i think this is very very different than how you all work where you validate that your findings are correct by making and convincing arguments to your colleagues and then if your colleagues are convinced then then that's like becomes like a new standard of of like what people think the world works so that's my understanding at least there's not really a concept that you can verify but this is exactly how things happens in the past um so that's basically all I wanted to

[29:03] say here. And I think, oh, I had one last thing I wanted to say. Our grant money is typically oriented around building new CS methods, and it's harder to get money for applying existing CS software to new problems. And so something that we all need to, us CS people need to argue when we're writing grants, this is not something that ChatGPT, or Claude have already solved. It's some new problem that deserves, or it's a new problem, or it's an existing problem that they have not solved and that we have some proposed methods to solve. And so what we're interested in, especially when we're like framing research that's going to get us tenure and get us papers is what can ChatGPT and Claude not do? And how can, as we as computer scientists, address this gap in the thing that they cannot do? And so I think like that was a really good example from what Anjalie gave from her research was that the ASR, automatic speech recognition records is bad. And that is like sort of identifying something that even the fancy models from

[30:04] these big companies haven't solved this. That's something we can work on and write grants for and get money for. And so you should, this is something that I talked to Michael about a lot, but you should always like when you're trying to collaborate with us computer scientists, see first, can Claude just solve your problem? And if Claude can just solve your problem, then maybe what you need is, Anjalie said is like an undergrad who is going to build you a good cloud solution. Maybe you don't need to collaborate with computer science PhD students. Okay, with that, I will turn it over to questions. Thank you. Yeah, so we thought we'd just do like an open Q&A here. In case there's awkward silence and no one has anything to ask us, we did like brainstorm some things that you could just read from the slide, if you can't come up with your own. Sher, do you want to start? This is really interesting. And I've been thinking a lot the last few days about vibe coding. And I was wondering what you feel like the role of that is in these collaborations,

[31:05] considering that it can sort of bring a lot of ability to historians do this sort of computational work. Yeah. And please restate the question for the audience. Yeah, great. So the question was, what is the role of vibe coding? And particularly given that that perhaps enables historians to do some of this kind of work themselves. Do you want to start? Yeah, I would say that vibe coding is really great. Because if you can do this work yourselves, you don't need to bring on an AI researcher. Because what we're really interested in is the things that are too hard to easily just vibe code. And so something that would be really helpful to us is if you have some problem, maybe you have some data you're trying to analyze or something, and you try by coding it and you can't for the life of you figure out how to get it working. And you've now convinced yourself that like, Claude is just too stupid to do this. That's when you should approach one of us. That is exactly when those are the kinds

[32:07] of problems that we're excited to get. I guess like the other thing I'll add is I think by putting is when you can verify the outputs and you know what it's supposed to look like. look at the output and say like yes this worked and it's doing what I wanted to. I have actually out of the project that I talked about the my second student Cole, he so we have this very specific example which is this was like six months ago we were building this system and he said we were doing this for Baltimore police data but in CS we have to show generalizability and so we also ran everything on a set of police radio communications from Chicago so we could show everything we did worked in multiple settings. And he had run everything in Baltimore, and then he switched the Chicago data, and it just didn't work anymore. And we had this whole hour-long meeting, and we were like, oh, maybe the data is more sparse, and what could be different about these cities, and why wouldn't it work? And there's theoretical guarantees, what could be going wrong? And then he came in the next week, and he was like, so I figured out the problem. Before I switched data sets, I asked ChatGPT to refactor my code.

[33:11] So not even like Vibe code, just like you know, refactor it into functions. And the method that we're using uses like a particular calculation. And what the model did was it added like an if-else logic that defaulted to just like the wrong formula. And so once he like found that bug and fixed it, then everything was working again on the Chicago data. But that was something where I was like, I think a lot of times these like AI code is really good. And humans make mistakes too. Like it's very like students write bugs all of the time. But this was like one example I had of I was like, wow, if like we had been doing this and you had not written the original code from scratch, like I don't know if you would have figured out what the problem was. I have two questions. It's just the role of benchmarking in the kind of two disciplines, how you guys think about that. And that was part of the implicit in the question. I don't know much about it in the funding of archaeology, but it

[34:12] seems like that would be a tension and a potential collaboration. And then the second is how you think about paper writing, whether you guys discussed student paper writing, but in terms you think about actually writing papers across disciplines. Why or why not? Do you actually steer toward projects that will produce CS papers and archaeology papers? Or are you actually interested in those? and how do you think about this? To repeat the question, the first question was the role of benchmarking across disciplines. So why don't we start that one? Yeah, I think in terms of benchmarking, this is very true. I mean, we've had a lot of conversations with Daphne and our collaborators who span sort of computer science and archaeology about humanity's last exam and these major benchmarks. And what we've been thinking a lot about comes out of some debates in archaeological theory about what's called middle range theory that a lot of ink was

[35:14] spilt on in the 1970s and 1980s. And I'm interested to have conversations about this with people in history to see if it's similar. But basically, you have low-level questions that involve low-level data things like how many archaeological sites are within 50 kilometers of Rome, right? many Roman sites are within 50 kilometer of Rome. And those are very like data extraction, simple questions. And you have huge questions like, you know, what are the factors responsible for collapse of the Roman empire, which is a huge historical question, which also a lot of ink has been spilt on it. But what we're sort of been thinking about and discussing is the middle range questions that sort of link the data to these larger issues. Like the big LLMs that are trained on books about Roman history, they can spit out, or at least it seems like they can spit out some of the major factors that have been identified for collapse of

[36:18] the Roman Empire. But when you ask them to link the data to those theories and sort of be a bridge between data and these higher level explanations, that's when they start to hallucinate and they make up data And they get dates wrong like I had querying them. It was giving me all these volcanic eruptions that were like 500 years off from the class of the Roman Empire. And I'm like, this is not even close, right? So we've been thinking about middle-range queries, right? And these sort of mid-range questions that involve – some of that involves going into, like Daphne was saying, databases. and sources of information that are sort of highly specialized, not open to the web. Can we benchmark models against those kinds of middle range questions? Because that, it seems, is one of the things they seem to be really bad at.

[37:19] I'm not sure if that's true for other disciplines. I'm actually not entirely true. That's true for archaeology. One thing to that, too, is we love if you can give us data that's not on the intranet. Because if a question can be, this is something we found at the existing benchmarks that we sort of audited, is if a question can be answered of intranet data, the model is just going to answer of intranet data. Computer scientists, AI people are really interested in what questions models will be able to answer on the data of the future, which is not in the pre-tribbing data of the models. and so like we Michael had a had a collaborator who maintains one of the big archaeology databases and he just gave us a tarball dump of of all of this data which has not been totally indexed by by the big by the big AI companies and so now we can like ask interesting questions um and test how good AI is on data but it didn't see during this training process which is exciting to us as a computer scientist.

[38:20] Yeah, so that's what we've been trying to benchmark, those sorts of questions against databases that's not on the internet. Yeah. And then the second part of your question was not on benchmarking. Co-writing. Oh, co-writing papers. Yeah. In the question of this database work, we haven't quite got to publishing yet, We were actually connected up through Schmidt Sciences that gave us some development funds for a pilot study. And so we're at the very early stages. That's how I met Daphne. And we got going on it and we put together a team. But it is, you know, I think that's one of the things that makes it challenging is the different disciplinary expectations, the different venues, you know, how we're going to manage that. it's something that involves a lot of communication and a lot of, uh, that's, you

[39:22] know, because, uh, we just submitted like an abstract to some archeology conference thing. And like, uh, Michael and our ever collaborator, Eric, and it asked me to like help with the writing. And I've literally never seen one of these before. So like, I, I don't know what, what writing I should put into it. And I think it would be the same if I asked you to help me write like an LLM paper. Um, yeah, this is, yeah. in computer science that's like vast, right? Is like in your discipline, submitting a conference abstract, like 90% of them get rejected. And in archaeology, I mean, pretty much everything gets accepted unless it's completely outlandish and ridiculous. So, and just the style of writing, the style of communication is very different. different. And yeah, so it's, it's a little bit challenging, but I think this is where some patients and some, you know, this is

[40:23] where putting students together, graduate students together from these different disciplines helps because they can push it forward, you know, into the future if they, if they understand that their discipline is, is idiosyncratic and not representative of, So that's a good reason for having students and graduate students. A joint grant was also helpful. Presumably, if you're applying to a grant jointly, it's a grant called that's relevant to both of you, and you have to sort of negotiate how you're going to write that. So I know I learned a lot. And in history, the narrative is analysis, right? Like that's the analysis. How do you prove something in history? will you convince other historians that what your argument is valid and accurate? That's entirely different than scientific fields, which have a correlation coefficient or have some numeric validation out

[41:23] the other end. I guess I'll just say, like, I can only speak for myself, but I'm super open to co-writing. I think my department also is pretty supportive of interdisciplinary research, which is not true of every CS department. I'm always happy to have some stuff that may not be specifically career advancing, and that's fine. Yeah, even for students, sometimes it can be a good experience, but there I think I'm just more protective of their time because they need to get enough done to have good jobs when they graduate. I have a question here, I think. Yeah. Working with some computer scientists, and you've had these kinds of conversations, so I'm learning how to communicate with them. want to ask them to make a pitch to them, but I don't know that I'm really sure what I'm talking about, or I know I'm not sure. So maybe you are. But the question is, I really want to create a multi-modal project in which multiple different sources are being

[42:27] coded and used in relation to one another to generate new data. And so I see the challenge for the computer scientists to be, I'm trying to bring them new data. It's not on the internet. Audio sources, textual sources, and visual sources. And I want them to code that data. And I see the value added being that this data is in conversation with each other. answer some of these middle range questions with the data in dialogue with these larger historical questions. Does that make sense? And if not, like, how can I explain that as a request? I'd fight. If you came to me with that pitch, I'd be like, let's chat about the data. Yeah, I think that's one of the things that, like Daphne was saying, the current LLMs, because our language models don't, you know, I could be wrong about

[43:30] this because things change really fast, but like some of them aren't that great at integrating all these different modes. And so, and this is true for archaeological databases. You have images, you have text, you have hand-drawn maps, you have video, you know, like you were describing audio. Yeah. So So that's one of the, I think you could, how you're explaining it would be appealing to computer scientists, right? Because that's one of the shortcomings that's out there is integrating different types of data together to address one historical problem if you're coming from a historical. The other good thing about what you just did for your pitch is not making it specific to any one data sets. because you'd want to convince the computer sciences that there are interesting generalized methods, which I have no idea what your data is, and that's actually good for talking about. That's interesting. I think my first questions would be, like, how much data do you

[44:33] have? Like, what are the modalities? What have you already tried with it? Because usually, like, volume of data is not, like, it changes so much what you can do, and so that's, like, usually the first question. And you get no sense, like, I have no idea which part is necessary. some indication of that? I think, so it depends what you're trying to build. If you have a small amount of data, these days off-the-shelf models, a lot of times, like you're basically looking at an off-the-shelf model. If you have like, let's say less than, I don't know, 50 to 100 data points, there's not a lot you can do to adapt using that much data. There are some tricks for that small, but not a lot. But for textual data, is there a number of pages or something like that? I guess you're dealing with audio in the example you gave.

[45:33] Yeah, it depends what you're trying to do. But these are just a guidebook, the parameters you would want to tell your computer scientist is, what is the input you want into your system? What is the output you want the system to produce? Have you tried getting Claude to do this and then does it fail? And then how many examples do you have that are already labeled with the output that you want? Like sort of like a gold standard label. And how many unlabeled examples do you have? Now, those are like, I know this sounds like really like boring and machine -winding, but that's what immediately goes through our heads. And then once you tell us that, we can tell you, okay, there are methods that are probably already going to work for you. this is an interesting research problem where we should put a PhD student on, we can sort of give you feedback along those, sorry. Question there? Yeah, so I want to circle back to the thread about co-writing, whether it's co-writing publications or co-writing grants. And one of the things that I always find invigorating yet also challenging about

[46:34] like radically interdisciplinary collaboration is that we either use different words to mean the same thing, or we use the same words, but they mean totally different things. I mean, even just like the basic, like, are we like being each other in some place, right? Often takes a lot of just like free conversation, free work, you know, collaborations that may be sort of founded before they begin, because we just are not sure whether we're on the same page or not. So I'm wondering if you can talk about how your collaborations have sort of dealt with that, like ways that you've been able to be successful at the level, even of that basic, are we talking about the same thing? If that makes sense. Yeah, it's an excellent, you're making a lot of excellent points. I'll rephrase the question. Well, yeah, I think hopefully I'm framing it correctly.

[47:38] But your question pertains to interdisciplinary collaborations and the use of jargon and language and how sometimes people are using different terminology to talk about the same thing or the same term to talk about different things. And so there's a lot of sort of learning and dialogue. And, you know, we ran into this with working years ago with statisticians. We started talking about archaeological survey, which is when archaeologists go out and do like exploration of an area. And the statisticians were like, oh, you mean like census data where you go door to door and interview people about their household? And we're thinking, no, that's not it's not really the type of survey. Like he's thinking about surveys like online polls, surveying people. And we're talking about something incredibly different. So we really, I mean, this is what enabled working with Daphne and is that she did undergraduate study in archaeology, right? So we didn't have to start with the basic, like, what is archaeology?

[48:40] Because she almost, I think you almost had a major, right, or a minor in archaeology as an undergraduate. I teach the master's students on the project what stratigraphy was on the board by drawing things from my undergrad archaeology classes. And she worked with a large team that was doing computational archaeology by a faculty member who's now at the University of Chicago. So she was doing computational archaeology and had that experience. And this is a perfect example of why training students is so important, right? Because then they bring that experience into hopefully becoming faculty members where they can speak across disciplines. But it's really, yeah, it's really challenging because you, you know, I think we had, when we had a workshop recently, we had a computer scientist asking us, I was having a hard time understanding what his question was. But his question was sort of related to, well,

[49:42] isn't your job as archaeologists to distill the ethos of an ancient society to understand their mentality and their ideology, right, and convey a sense of their humanity or something? But from scientific archaeology, that's not really – that wasn't really our objective. So even, you know, just understanding what is the overall mission of a discipline, I mean, that's something that takes a long time to explain and can be really difficult. So, yeah, so there's not really easy answers to that. I think, Anjalie, you had some background studying classics, right, which sort of enables you to bridge that, right? But a lot of people, this is the problem with increasingly training students in hyper-specialized fields is they don't come away with some basic understanding of some fields that are

[50:42] really distant from their own. Yeah, I don't have a general solution. I think just like you have to keep talking to each other. I have like two recent examples of this. So one was writing a grant with public health folks. And here, I guess, maybe some of the messages I think what we try to prioritize is writing to the audience. So whoever is like closer to the audience can kind of take the lead on the terminology. So we were writing a grant on a kind of joint initiative and they use the term like we're going to build a surveillance system. And I was like, in my field, surveillance has really strong negative connotations. And this is a standard thing that people say in public health, but I think this is also going to have AI readers and like we need a more neutral term. And then I had this as well with Steve Riss, Sociology Collaboratory. The first draft of the paper, we like use the word like data sample in a couple of places. And he left us this long note on the draft that was like, OK, I think I

[51:44] figured it out. Like by data sample, you just mean like an individual data point, like a single transcription. Like when we talk about a sample, you know, we're talking about like a sample of the population and is this a representative sample? And it was just like literally a terminology difference. So in those cases, like we did just change the terms to be ones that we thought would be clearer to everyone in both of those disciplines. But I think like, you know, like we wrote the first draft on that because we're sending it to a CS venue. And I don't think any readers are really going to have like if we had left it as sample, it probably would have been fine. but like the input from Steve helps us make this more readable to people from his field. Yeah, so I don't know. I think if one of you gave me some data right now, whatever you're currently working on for your work, and I started talking about examples, like how many examples are in it? Let's train on the examples. What do you know what I'm talking about? Or is that weird? This is like super standard machine learning terminology where like an example is a data point. And a data point can be a book, can be a page of a book, but we

[52:49] don't care. We just want to frame things as these are separate, independent examples, which is not how you guys use examples at all. It's like how basic. Yeah, it's a unit of data. But I think data in itself for historians, I think many of us who aren't in computational history would think of something like statistics and not a corpus of text as data. And I think that's what we kind of have to struggle with wrapping our minds around for people who are in cultural or intellectual and not coming out of... As a CMU example, the CMU library graciously gave me just a tarball dump of all of the digitized records of Herbert Simon, his famous AI researcher from CMU. And when he died, a lot of his works were donated to the library. like his notes, his personal records and stuff and digitized. I give this to my student. My student now doesn't care at all about any of the

[53:50] things you would care about. These are just examples. Each of these documents is now an example. And he is going to run OCR on each example to get like the text in it. And now he is going to build a retrieval augmented generation system that retrieves examples and then generates an answer to your question based on them. example in this case is a document like a single document from the archive but we have now dropped any sort of like I think in the computer scientist's mind there's like no like there's nothing distinguishing these from each other there's nothing special about one or the other they are just one of many things in this repository of examples that that's very much how our students think. And it's honestly like you guys are very, very used to looking at the documents that you work with. It is like pulling teeth to get our students to look into any data set of them.

[54:50] That is very, very hard, whether or not it's history or however it was. There's one more question over here. I'm going to take the last. Yeah, this is, I think, I'm a diplomatic historian in the 1990s. This is kind of a personal foom of like agents not existing. I had to transcribe each page. You know, there's a trend that pages hand by hand, you know, one of the times now agents running 24-7 is pulling things out of the NARA cloud and kind of building a giant surgical corpus. The challenge I have in kind of history meets computational history or computer science is very often the output is this kind of tidy word cloud or tracing mentions of a certain phrase over time or even in a natural language cluster but it's often the following format. I've looked at UN reports since 1990 and the prevalence of the words humanitarian or humanitarianism has an uptick since 1991. Ergo, more humanitarianism. I'm like, this is kind of epi-phenomenon, right? So I guess the challenge I have is someone looking at like decision points on when the US and its NATO allies chose to intervene in some massive frosty situation.

[55:51] That's my variable of interest. I'm looking at like three or four peoples like Bill Clinton, Madeleine Albright, Tony Blair, like them alone, right? other people and even a vast corpus but it only matters if it gets to the very top so what i'm trying to use the computational methods for is to zero in on like literally maybe six or seven documents of critical moments so there's a meeting at the white house that really changed the needle for a few top people and as far as i can tell this is like not really what the kind of computer science meets historian because it is really about building this data set where the the regression you run on the data is the output. And I'm wondering if this kind of collaboration, we are using the computational methods, but to triage within, again, a potentially infinitely large, not readable by a human corpus of millions of emails, millions of pages of data, and zeroing in on a qualitatively analyzable, that is, you actually read a few hundred, a few thousand critically important pages identified with the machine reading methods, so that you can tell a qualitative narrative

[56:56] that I'm trying to do, but without a computer scientist like you guys to advise me. So I've been vibe coding up storms since May, but like iterating, iterating, iterating, and this has been amazing. But is it the kind of thing you'd like to collaborate on? Or is it too alien because this is qualitative narrative? So the question here is this sort of needle in a haystack question. A historian cares about six documents out of like a huge amount. And this is actually, I think you could convince an AI researcher to work on this, but you would need to frame it as Even your fanciest search tools can't handle the kinds of queries I have and being able to find the one or two documents I want. And if you frame it to the AI researcher as you guys think you're so smart, you think you've built so many good methods, but those methods actually suck. And I have an example of them sucking. That is something that you will get buy-in for. and better yet if you have like 10 kind of examples of of search sucking uh or like so like more than just one basically like this is a trend they cannot

[57:57] handle these types of queries this is kind of the origin of the collaboration i have with michael too it's like we were like trying to do these middle range queries of this archaeological database if that's something that i think you will get an ai researcher to bite on this so i mean well thinking but the ai people are doing critical thinking. That's exactly right. They're not. Yeah. You provide them evidence that they're wrong. And the trick there is too that I think well from a humanities perspective let me just say that I think the issues humanistically are deeply important what you're talking about. Intervening in humanitarian crises and what what that prompts major national, international powers to sort of intervene in that, right? So there's a very deep humanistic importance to that problem. And like Daphne's saying,

[59:00] you know, it's hard to put your finger on, like, what is it that makes those particular documents important, right? That's what you, you know, explaining that the current models to a computer scientist are failing. So what is it that would make a historian pull out that document out of the pile and say, aha, this is what I'm looking for, right? That's not always easy to explain because you don't necessarily know what you're looking for, right? You're looking for an anomaly that you haven't found yet. So in our database of archaeological objects, we have, I think there's a few hundred thousand artifacts. And one of the approaches we were thinking about is, can it find the anomalously important or the unique or the idiosyncratic things out of there? There's a name for that, right, Daphne? Detection. Anomaly detection. Yeah, we had a bunch of discussions about, are we

[1:00:06] looking for broad general pattern recognition, or are we looking for anomaly detection? And this is what the computer scientists were trying to prompt us. Like, what are you looking for as archaeologists? And oftentimes you're looking for both those things, right? What you're describing is sort of an anomaly detection issue. Yeah, and it's a great explanation, right? If you can convince them that, you know, as powerful as these current models are, they're really bad at this. And this is a really important issue. You know, and explain to them, like, what are some salient characteristics that you're looking for? I mean, when we started to do computer vision like this two years ago, we were started to think about like when a human looks at an image and it finds things, it says that's a person, that's a car. You know, this becomes very important in self-driving cars and things like that. But with idiosyncratic patterns, it's not always obvious why

[1:01:08] humans, like they're making sense of images in complicated ways. They're seeing things and recognizing patterns that are sometimes we don't even understand or can't really explain why we're picking out anomalies that we are because it's so intuitive, right? So that's also a critical part of it is pinning down like what constitutes the anomaly that you're looking for to help program something to do that. I think we're running out of time, but I'm happy to talk more about this. My lab has some work in this space. We put out a paper about a month ago on data selection for like downscaling for analysis. And then we had a paper last year on corpus analysis that tries to do like text analysis more similar to the way people do qualitative analysis. But I don't have like a gold solution. Like the reason we're working on it is because it's not solved. But yeah, maybe we

[1:02:11] can talk about this later. I didn't know there was one more question. I feel free to come talk to us. Yeah, absolutely. Have a break. Yeah, even better. All right.