Abstract

Clifford walks through the pipeline behind the Imperial Careers Atlas, which maps the careers of British colonial officials, instead of running a hands-on workshop. The pipeline starts with optical character recognition (OCR) of the biographies printed in the Colonial Office List and the Indian Office List. A Python script splits the text into individual biographies and an open Qwen model converts each one to JSON, after which Python rules fix dittos and merge duplicate people. He then grounds places and people in Wikidata using an MCP (Model Context Protocol) server with vector search and QLever for deterministic matching, reaching 86 percent grounding for places. The discussion turns to cost and to how clean a dataset needs to be.

Chapters

Transcript

AI-generated transcript. It was produced from the recording by speech recognition and cleaned up with AI. It has not been edited by the speaker and may contain errors. The captions on YouTube are AI-generated in the same way. Each timestamp links to that moment in the video.

[0:11] I imagine we're going to have people joining us as they leave the lunch, but we should probably get started because they've only given us an hour to do these presentations. So this was billed as a workshop, and then it occurred to me that we only have an hour and there's no real way that we can work with a dozen different datasets. So I thought the most productive thing I could do was to do with the data sets. do was just walk you through how the pipeline worked that made the Imperial Careers Atlas possible and sort of talk as we go about how it can be applicable to a wide range of other historical projects. The core process of grounding entities in primary sources to Wikidata is kind of the thing I'm hoping you can take away. And I also didn't really know what my audience was going to be like, so I'm sort of

[1:13] going for intermediate here, not pure beginners. So throw up your hands. I've tried to make sure that there's information on slides that glosses some of the technical terms that I'm going to be skipping past quite quickly. and these slides and the websites are all available and they'll stay online. Okay, so well the first nine months of this project I was trying to extract different information from this document. In a conversation with Claude Fable in May, it pointed out that why don't I just use the biographies that are at the back of all the issues after 1883. Upon that realization, I'd found them in the Indian Office List, and I thought they were not there in the Colonial Office List because I must have checked the year before they started including them. So it's just these very short biographies for every

[2:14] mid-level or higher colonial official. The Colonial Office List is everywhere except for what's now Pakistan, India, Bangladesh. and then there's a separate set of similar documents from the Indian Office that provides biographies for every official in the Indian Office. This is just the bureaucratic divide in London. So by extracting all this data, I was able to make a fun video that went viral around on the web showing the movement. And this is actually the question I kind of started with. I was interested in how many people who have a colonial role in Canada end up in other parts of the empire. Because we tend to think as Canadians that we're not really a part of that part of the British Empire. We're the settler colony part. But in fact, lots of people who have high-level roles and training in Canada end up

[3:15] as the head of the Raj in India, for example. So there's lots of movement, not just before Confederation, but also after Confederation between Canada and the rest of the empire. But as I've sort of noticed with AI, it's often easier to just do everything than to try and scope the project down to the narrow question I had. You know, once you're trying to extract the Canadian related colonial officials, it's much easier to just extract every colonial official and then ask which ones are connected to Canada. a sudden you've created a project that's much bigger than the one you thought you were creating i am not a historian of uh officialdom in the british empire i just created this atlas sort of by accident as one does nowadays so the arc is just these you know basic strings i'm not going to talk about ocr there's been a bunch of other panels on ocr one thing that makes this possible so much better than it

[4:20] was before vision language models. So now we get very clean. We will see some of the errors that still crop up and the problems that they caused for me. But most of the time you get a very clean extraction of the text that's one for one from what was on the page. You can see it's very truncated, lots of abbreviations, and it's structured but only semi-structured. So that's going to cause some challenges. But, you know, a human could quite easily pull out the person's name and the different positions that he held in the year that he starts those different positions. Once you have that, you just need a latitude and longitude for where those positions were, maybe the capital city in each of those colonies, and a date, and you can make an interactive visualization.

[5:24] Now, as I mentioned, there were two sets of documents. So this was one of the challenges is that a minority of especially high-level officials do move back and forth between the Colonial Office and the Indian Office. And so I had to make sure that I wasn't counting them as two separate people when, in fact, it's the same person just showing up in my two source collections. Obviously, when you have a birth date and a name, if those two things are extracted correctly, it's very easy. When you don't have a birth date and the name is ambiguous or common, then it creates new challenges, especially when you're trying to do this fully automated. And I probably should have said this at the beginning, but this project didn't have a grant and it had zero RA hours. It was just something I was doing to figure out what you could do with Claude Code.

[6:27] So I'm still trying to figure out the answer to this slide, right? And at some point, I need to connect with some historians who actually study the kind of questions that this data set now helps answer. But obviously, there's a lot in these records. They're really dense and packed with information. for quite a few of the officials. We know what high school they went to, in addition to what university they went to, when. We know what imperial honors they were given, which is a big thing for the British. They like making people into knights and all kinds of other honors that are connected to particular imperial identities. It gives us all kinds of information about movement, but also they don't show up on the map, Also, the people who don't move, the people who just go and work in one colony at a mid-level police commissioner or something like that, slowly rising through the ranks. But they're in Ceylon for 30 years and go back and retire in the English countryside somewhere.

[7:31] Wikidata is crucial to this project, and it's really something that has unlocked what we can do as digital historians. 15 years ago, I was a part of a project called Trading Consequences, where we tried to do something similar, where we used the Edinburgh Geoparser to identify all the place names and then ground them to GeoNames. And back then, you had all kinds of failures because GeoNames didn't have things like the Straits Settlements. I don't think it had Ceylon. So all of these 19th century toponyms weren't there. And we kind of imagined that to make this kind of project work, a big half a million dollar grant just to build a gazetteer, for those of you that were in Ruth's session, that would allow us to ground entities to all of these British world locations that don't exist anymore. Well, in the preceding 15 years, you know, the people of Wikipedia have just done that work. And then Wikidata has extracted that into all of these unique IDs so that we have

[8:38] Wikidata unique ID for the colony of Ceylon that has all of its basic information you know the year that the British take it from the Dutch the year of independence the capital city it's uneven because it's Wikipedia-based projects that some entities are really rich and others just give you a unique ID but really if you have a unique ID and a latitude and longitude that's enough to build from for most of what we're doing as historians. Any questions at this stage? Yeah. I guess we're creating some kind of API that does plug into that data. I've got slides that are going to show you the MCP server that's made it much easier because it's harder to do if you're just trying to use the SPARQL queries. It breaks all the time. But there's also an alternative thing called QLever that

[9:41] doesn't break, QLever. I think I have it all on the slide, so I'll move on. Okay, so this is the basic pipeline along the bottom here. So the OCR was the first step, and that's months of my time. At first, I was trying to follow Mark Humphries. I realized if Mark Humphries could do handwriting, we clearly could do 19th century government documents as clean or better. But that's expensive to run all of this through Gemini. So after kind of playing and learning in the summer of 2025, I realized that my university has an underutilized supercomputer with a bunch of A100 GPUs that the physicists don't like using anymore because they're five years old. And I could just log on and get like three days of access to a

[10:41] GPU. And that there was these other projects. olmOCR that Allen AI created. And it's basically these AI companies that were trying to create open LLMs to compete with the closed models. They needed to get better training data. The Internet Archive has half a billion pages or something like that digitized, just sitting there free to download. But you don't want to take their OCR and train an LLM off it. You need cleaner extraction. So they were building things that were kind of fast and accurate enough. It turns out olmOCR was really optimized for speed, and it strips out the headers from every page and the footers from every page. And that gets really messy when you try to then reconstruct anything. But in the winter, Chandra 2 came out, or maybe November last year, Chandra 2 came out,

[11:43] bounding boxes and like tells you where on the page the information is it does two columns just fine and it has an accuracy level of like in the high 98 90 percent 99 percent for 19th century text so just completely unlocks this kind of work the next is to try and extract it as like structured data into a relational database a JSON knowledge graph whatever knowledge schema working with. So you're pulling places, events, people, giving them their correct entities. And then you had to clean that, figure out all the different kinds of mistakes that your pipeline creates. I keep finding them. Ground, hopefully, to Wikidata. And then once that's done, you can have all kinds of fun with visualizations. And these models are fantastic at visualizations. that video I showed you is like an hour's worth of work. You give it a clean data set and it can create these visualizations incredibly

[12:47] quickly. So you met the connection to the Wikidata. I'll get there. Still got 45 minutes. Okay. So here's some of the challenges. You know, the OCR is very, very good, perfect I'll show you some examples of that but we also have annoying things like dittos and like everybody who's worked with historical sources has run into dittos right so you don't want to ground a bunch of your locations to whatever Wikidata decides there's probably some spot on the planet called ditto we could have like a giant ditto like so many careers going to this one spot, right? Okay, so here's the brief little exercise. If you've got a computer in front of you, you can just put in this. If you've got a phone, you can use the QR code.

[13:49] I should not go forward before you've had a chance to use the QR code. You can just take a minute here to be a research assistant in the pre AI world. So we can think about what it takes to structure the Balmer biography It's a bit finicky on a phone. There's no final exam in the class, so don't get it working perfectly. Not work. Imperial careers workshop backslash annotate.

[15:05] Is there a birth here? The last two questions. What is the last two? I think he's at the post and telephone service. 1933. 33. Where's the location? Does it actually work?

[16:05] So I said that this would be the old-fashioned way to do it. This is also required if I ever want to get around to publishing this. I'd need to create a gold standard and get students to create 250 or 500 human annotated biographies so that I could actually figure out exactly what my recall and precision are. I haven't got around to that yet, but it's on my to-do list. So you build something like this, you annotate it. You can see it's a pretty cumbersome process. It's not actually that easy. even as a human to read this, you'd have to get familiar by doing hundreds of them to really figure out how these abbreviations work. And it's actually a place where the LLMs have an advantage because they can read a thousand biographies in a second and start to notice all the patterns that are in and figure out what the abbreviations actually mean.

[17:07] Yeah. it does yeah so i'll get to that in a second too so this is the actual process i should have started by getting students to create a gold standard i didn't have students didn't feel like making a gold standard myself so i skipped that step. I just took the clean OCR and said, Claude, how do we turn this into a knowledge graph? And the obvious first thing to do is to see how far you can get with Python. Because if you can write a Python script to extract them, the biographies cleanly and parse them, then it's all deterministic, it's safe, it's easy, it's cheap, it's fast, it's free. And I got quite a bit done with Python. So I think we were able to write a Python script that splits the just stream

[18:09] of raw OCR into individual biographies. It was consistent enough, the patterns that you could figure out where one biography started and one stopped and turn it into a JSON file with all of the hundred thousand biographies as a stream of text instead of just the full pages of streams of text are these printed or printed yeah okay but as i found with all of the projects that i've worked on regular expressions so using Python to identify patterns in the text, it works until it doesn't. And we have two factors. One, most of our sources were not designed by people with computers in mind. And so they're just, they are regular, but they're always semi -regular. There's always places where the rules break.

[19:09] So there's like patterns on almost every colony in these collections when I'm trying to parse the actual core part of the document, after the name of the colony, it says like geography and climate or something like that, except for it doesn't say that for Canada, Australia, South Africa, and Jamaica. And so then you're trying to create a list of rules. Okay, in that case, you have to look for this pattern. It always breaks. It always gets to the point where you can't create enough rules in Python to do it in a deterministic way. So you use Python as much as you can with the help of Claude Code. How much can we do with Python before we start having unacceptable levels of errors or errors that are going to be more trouble to go back and clean than to just use an LLM for the parsing? When you ran the OCR, you was an OCR on purpose. So here is what I want to do with the output text.

[20:13] No, just a straight Chandra 2 output. You could do that with, you know, if you're using Gemini or something like that, and you just gave it these biographies, you could ask it to extract the biographies into a JSON and skip that need for regular expressions. But then you're relying on it to not hallucinate or do something weird along the way. think in general it's probably safer to just get it to do its one task unless you're using like a frontier model and have an infinite budget because i would trust opus 5.5 but um yeah that's just becoming you know you're using a sledgehammer when all you need is a small a much smaller screwdriver or whatever when you're using those big models and you're using more energy and cost than you need when you can just do it with Python Okay, so two irregular for regular expressions alone.

[21:13] So here's just a few examples. We saw when we did the little exercise that there's years all over the place. And regular expressions are very good at seeing that pattern. So you can write it right here. It's going to find you every date in the 90s. And you can have that for each decade and extract all the years. And you can confidently say this is a year. But is it the year of the birth? they changed positions? Is it the year they graduated from Cambridge? Then you have to layer in more rules to try and figure out that pattern. And if every biography followed the exact same structure, then you could do it all in Python. But the biographies don't. They have enough difference. Some of them have births, some don't. Some of them have education, some don't. They don't have like a nice clean like semicolon is only used to separate career movements it's too irregular so this is where we had to come up with or i had to come up with the hybrid approach

[22:23] so at this point we use Python to do the original split and then we used an open model it's the Alibaba family of models called Qwen that you can run on again that local cluster or even on a decent gaming computer or if you have one of the nicer Macintosh Pros you know I think 32 gigabytes 48 gigabytes of RAM is going to be enough to run this model if you've got a GPU with that kind of capacity. It's also very cheap on something like OpenRouter. So for dollars, you can do this work on one of the sort of cloud platforms out there. So we use Python to split each biography, and then we gave Qwen just that individual biography and asked it to turn it into structured data.

[23:24] So this is an idea of the different ways there's been a post six or seven months ago on Louis's Substack just on how good it is to make friends with whoever controls the cluster at your university. There's a chance that there's a cluster that the scientists have stopped using because they want the more advanced hardware they want H100s and so if there's some supercomputer sitting underutilized, figure out how to access it. This used to be really, really hard. I did some work on a cluster in 2021 or 2022 where I had access to a coding model by sort of cutting and pasting the code into the chat bot, trying to figure out how to get jobs to run in a cluster Linux environment. And I'd get it to work, but it was a huge headache. Now with the agentic models, you can just get them to SSH into the cluster, set up the entire job, upload all your data, submit the job correctly,

[24:28] watch for it to be done, and download it back to your local machine. So this is, you know, the power of these agentic tools like Claude Code is that they can do this stuff really, really efficiently. and it takes that barrier for those of us who weren't trained in computer science and don't have staff to submit jobs to a cluster, you can often get this to work. Again, if you've got the right kind of Macintosh or an NVIDIA Spark that you have access to at your institution, they run things much slower, but locally and for free, you know, for the cost of whatever the electricity coming out of the wall to run your Macintosh. And these models are enough that you can run them on a $3,000 Macintosh laptop. It doesn't take a $10,000 machine to do this. And some of you might have access to national infrastructure. So if there's any Canadians in the room, we get a lot of supercomputer for free

[25:29] if you sign up through the Alliance. I don't know for the rest of you what kind of regional or local uh supercomputer access there is but some of these places actually have mandates to like spread it out they want to be able to tell their funders that humanities scholars are using their tools so they get really excited when they come to you because it's like another checkbox that they can tell their funders it's not just the physicists we're you know reaching a wider part of campus so you might be surprised the the level controls the clusters around you. Any questions about that? Again, this is the route that if none of those things sound possible at your institution, there's something called OpenRouter, which is just a Cloud provider that has almost every model that's in existence right now. You put in your 20 bucks and you can submit jobs

[26:31] through their API. expensive anthropic or open models through them, but you can also submit it to these really cheap Qwen or other Chinese models. And there's now a few North American, one European open model. They're all much, much cheaper. So, you know, this would probably be 15 bucks worth of work, not 500. Any questions about that? So this is sort of the iterative place where you work with your coding model to figure out which prompt works well. And you can kind of cheat here a little bit by getting one of the frontier models to create your gold standard for you and look over its results and see, yeah, that looks pretty good. You skim through it. And then you test it against a bunch of the cheap models

[27:34] and you fiddle with the prompt until you're getting results that match the expensive model and then run it cheaply. so I got Claude to do the extraction for a sample subset and then we spent a morning using the OpenRouter because it's really quick and figured out a pipeline that actually seemed to be working quite well and once it was all working then I switched over to a machine I had control over it ran overnight and I woke up the next morning and it was all done so um we're basically just telling the model to go through uh very clear instructions only give us what's printed don't make corrections we don't want the model fixing anything for us uh we're trying to get it to something we can map so give us you One place, one event, a date, a place,

[28:36] and an event. Only give us the birth year if it's printed. Don't just guess. And then you audit the results. So you're relying on this cheap model to do this big scale extraction. and then the work comes in the next week where you just keep examining it, looking for all the things that broke. And then you can use Python in this second stage to write rules to go and fix the ditto problem, for example. Yeah. This is a little bit about the PUCASide, but I was wondering if you used any kind of structured output framework or you just kind of get it up and then structure it with any... No, it's all coming back to me in JSON. So it's got a structure and a type, and then I can build it into a database

[29:37] from there. Yeah, so we're getting the Qwen model to take the string of text and give it back to us as structured information. okay so early on this just involved asking Qwen or sorry asking Claude to go look for obvious errors I think this is one that it caught it just went through the places and looked for weird outliers notice there was a lot of dittos and then puzzled through how to solve that The second step was to build the visualization later and just look for everything weird in the visualization. And when you notice one weird thing, you can then get Claude to go and see how many other similar weird things have happened and try and create a rule that will correct a thousand mistakes from the one that you notice. But we'll get to that in a minute.

[30:41] okay so in this case it's pretty straightforward just go find the previous place name in the biography and replace do with that string of text and now we've replaced all those dittos in the data set and the important thing is that i try to always get the model to like record as much as we can. I tell it that we're going to write a paper about this someday. I have no idea if I'm going to write a paper about this someday, but I'm trying to get it to think like an academic so that we can trace each of these decisions. So when I get asked to give a talk like this six months later, I can remember what on earth I actually did back in June. And so encouraging the model to take notes as it goes in the data so that you can figure out what on earth we actually did.

[31:44] So I think we can see that this is in order, right? The biography. So when we get a ditto, it doesn't really matter where it is. It's just telling you to go to the previous thing. So it's already been extracted into JSON. Yeah, so the previous thing that had been extracted as a place, which is this one up here, it went and said, okay, the previous place from the position before was here, and we'll just replace ditto with that. So very simple for a human, right? We just automatically make that connection. But with a computer, you actually have to make a rule, test it, make sure you don't somehow put place into a bunch of job titles because the ditto could be for a job title in a new place. You have to be kind of careful, make a rule that's going to only replace what you want it to replace. Then you can run it across 100,000 biographies and get the fix for every single ditto.

[32:48] All right. So looking at this, do we think this is the same person? Because this is where the, you know, 98% OCR is great, but that still means there's 2%. And it's worth noting, I have biographies where every year it publishes a biography for the same person. So if in one of the years the OCR gives me a slight variation on the name, I've suddenly turned that one person into two people. And I have potentially two people with the exact same career path, except for one might stop short because I only captured that person in one year and then the other 39 match, right? So I think I've given a hint on the answer here. This is clearly just an OCR error where we've missed the C and the R. And this is the hallucination thing, right?

[33:49] It went for something else. When it didn't read the C correctly, it overcorrected to give me another last name that looked plausible. But we can see everything else here is matching very clearly. So we can write a rule for that. if you have a person where a bunch of variables match and the name is within a certain distance, it's probably the same person. Let's just collapse those into one unique ID. So, this is the one and one of the other also. That's it. So, you wait to the one that you have a bunch of. Yeah. But then that same rule can flag somebody like this, right? Yeah, Thomas Becker, Thomas A. Becker. And so we had to pay attention to all the other variables really carefully. In this case, it's pretty obvious based on

[34:50] their date of birth. Now, again, that could be an OCR error, right? So you had to watch out for eights and threes and things like that. But in this case, there was enough other information. They went to different schools. Nigerian Telegraph Engineer, the others a Victorian judge. This is clearly two different people. And so we kept them apart. Okay, so here is the fun part, which is trying to then ground it. So we've got to the point where we've turned the string of text into structured data. But we still just have a bunch of strings for place names for people for schools. It's not something that you can put into a map yet. Now we have to try and figure out which place in the world does this place represent. And you can imagine that in 19th century texts, Victoria is kind of a dangerous one. She's a queen for a really long time, and they name a lot of different places

[35:54] after her. Okay, so as I mentioned earlier, we rely heavily, or I rely heavily on Wikidata. Ruth gave a great presentation on the World Historical Gazetteer project. It's got a bunch of advantages. Wikidata has the advantage of coverage at this point, although Ruth's project sounds like it's about to explode and have way more coverage. But at this point, Wikidata is your best bet for finding a historical place and your best bet for an entity for places that no longer exist. or have been renamed somehow. Most of the time, that's still going to show up somewhere in Wikidata, and it's going to normally give you a latitude and longitude. Okay, and then the other bit of jargon here is model context protocol. And around a year ago, the German Wikidata project

[36:54] created an MCP server for Wikidata that is designed to have things like Claude Codes go and query it. And really importantly, it's a vector database. So it allows you to search not just for string matches, but for semantic matches. And that has proven to be much, much more accurate than just trying to do a keyword search across Wikidata, which gives you a huge amount of noise. When you're searching in a vector database, you can provide context like imperial official and get a much higher chance of getting the kind of match that you're looking for so you'll see along the bottom that we get different levels of success so it's remarkably

[37:55] good at giving clean matches for something like n road which gets expanded out to rhodesia seychelles so you can see with you know quite a bit of work on my part and lots of sort of iterative building we got it up to 86 percent grounding to Wikidata And then there's been a subsequent few months of finding weird errors or just things like Prince Albert, Saskatchewan, 100 miles north of where I live. Why are there three officials in Prince Albert, Saskatchewan? That seems suspicious to me. Looked at the three officials. They all were based in South Africa for the rest of their career. and you know a quick google later there is in fact a prince albert south africa as well and so you go into claude and say i think we got this one wrong and it goes and finds the examples rights rule fixes them people

[38:58] not really surprising here most of these people are not famous enough to have a Wikipedia page they're not published authors which is another way that you find yourself into Wikidata. So they're just obscure British officials. Oddly enough, probably half of these people are in Wikipedia because they were Cambridge or cricket players in their young adulthood and became well known in Somerset cricket or something like that and have a Wikipedia page as a result. And I was able to ground them to that. So, you know, military officers with some first world war prominence uh people who become like very high level officials in the british empire and cricketers are the ones you can actually find in wiki uh Wikipedia and Wikidata uh but this still means there's a big challenge in figuring out for sure whether i have individuals or if i've still got more uh careers than

[40:01] there actually were individuals I've done a bunch of work to try and make sure that I don't have duplicates, but there's probably still some duplicates that have been missed with OCR errors. Much higher success with educational links, in part because many of them just went to British public schools, and there's not that many of those. They're the private schools in England, oddly enough. Went to a handful of key colleges and universities. these are all the old schools. If you went to school in Jamaica, most of these places have Wikipedia pages because they're the prominent school in Kingston, Jamaica from the 1920s. I was able to ground most of them. This is what it looks like when we're using this MCP server. This is really a game changer. We were having lunch earlier and you were How well does this work?

[41:04] And if you're just trying to do like string searches or trying to ground things to GeoNames, you still get a lot of false positives and errors. But by switching to a vector database, the problem of grounding to Paris, Texas, which happened to us in Trading Consequences all the time, no matter how many times we thought we'd fix it, we'd run the whole thing again, and all of these Paris, Texases would show up when we'd map the data. really reduce that. It's much, much better when you have a model searching in a vector space and then choosing from the list of candidates with the context of where the information came from. You can normally get it to ground to the correct place. Can you set other parameters like the configuration of private design or also you just waited for the province in place? yeah um i think i've got a slide that's going to show us how this works okay

[42:05] sorry that's the q lever slide i'll come back to that in a second i don't have the yeah i was going to do a live demo of this and then it wasn't working correctly i gave in the website instructions on how to do this. Yeah, so you can add, so we could add like the word imperial or something like that. And you can test different parameters when we're looking for people. Then, you know, especially somebody with the last name Smith, you're definitely get better results if you say colonial official or something like that. But you can test and see what gets the best results. Okay, so QLever is deterministic, and it's a good option if you have something like a birth year. So you're able to ground a lot of the officials who are in the database, because

[43:07] especially the Indian Office List gives birth years most of the time. And you can pick from the different possible matches for Logan quite successfully. put in that he was a governor, that he was in northern Rhodesia. If he has a Wikipedia page, that information is going to be in the structured wiki data, and you can get good matches that way. okay so yeah i just wanted to give you a sense of the scale of the cost if you don't have access to a machine this is still uh you know by comparing it to how much it would have cost It's a fraction of a percent in terms of getting a cheap LLM to do it.

[44:12] Okay, and then as I mentioned, one of the things I found really useful is just putting things into visualizations and looking for weird things and then sharing it on the internet and asking everybody else to look for weird things. I knew Victoria was going to be a problem. So in that case, you had to build sort of special rules for these particularly challenging examples and say when the place name is Victoria, we're going to have to look at the rest of the biography to try and figure out which Victoria we're likely talking about here. so by looking at the rest of his biography he's obviously a canadian official with other ties to british columbia and that allowed us to say okay in this case he's going to be at the canadian victoria if somebody had been clearly in new south wales or melbourne or somewhere then you're going to say, okay, they've moved to

[45:15] the Australian Victoria. These are the edge cases that are harder to see. What counts is something that should be put into the map. And I think I just fixed this one as I was putting the slides together. We kind of reading through some biographies like, oh, I don't think that actually should have been mapped. When you're defeated, that's not taking up a new position. That's a position you didn't get to take up. We found other things that were getting mapped were like attended the Ottawa conference. Well, okay, this is not somebody who's a colonial official in Canada. That could be something else we'd map separately. How many of these people attended a conference somewhere and how far were they traveling for those conferences? But in terms of their career biography, they shouldn't have a very short jump up to Ottawa and then back to the Caribbean.

[46:16] So I think hiring some RAs at this point and getting them to look at hundreds of biographies will help me continue to root out some of the false positives that I've created by doing it just fast and cheaply. but we also just have this sort of old rule of big data that the overall patterns are going to be correct here so it really depends what kind of question you're asking from your data set i guess i've created something accidentally that genealogists and people like that might find interesting and then there's more pressure to make every biography correct whereas if you're just trying to see the big trends in the geography of the british empire then you know if five percent of the data is kind of messy it's probably not going to change the trend at all and you don't have to worry about cleaning so it's always a decision with these projects of just how clean does the data set need to

[47:19] be for the kinds of questions i'm going to ask yeah so these are some of the ongoing things what's missing what's hiding i probably haven't cleaned up the data of the people who stay in the same place because they don't jump out at me as errors there's probably more errors in the people who are in Ceylon for their whole career than there is in the people that move a lot because that's the data that i've been playing with and having fun with and other people have emailed me about so as we create these projects we kind of create new problems for ourselves and are going to have to puzzle out new ways to think about how do we get the data to the point that it's good enough for the questions that we're asking i think probably got to the point where i can end my slides and just take questions or open up for discussion for the last 10-12 minutes that we've got

[48:20] here yeah did you contribute the biographies to Wikidata uh yeah that's an interesting question they don't make it that easy um you know in terms of some kind of bulk way that I could just ship it all back to them. I don't think I can just contribute 25,000 people to Wikidata. If I wanted to work on this data with Susan Brown and her team at Guelph with the LINCS Canada and get it all to like a high standard of linked open data and publish it as RDF, then at some

[49:21] point Wikidata would probably be willing to ingest it. So again, that's just sort of a question of, is that the project I want to spend the next two years on? No, this wasn't, I've already gone way further with this project than I ever meant to. So that's probably not where I'll keep going. But I did publish the entire data set on GitHub. So if anybody in the room is really interested in Imperial Careers and wants to just take this away from me, you're welcome to it. I'm not hiding any of this data. It's out there for others to use. I'm just thinking that I'm in a different sort of discipline, but I'm just thinking of wanting to ask you, what are the most surprising insights that you, you know, arrived at as a consequence of working with this data that you didn't think that were, that you wouldn't have realized going in? I think the scale of the movement surprised me. You know, I'm a British world historian. I follow commodities more than imperial

[50:22] careers, but I know quite a bit about many of these colonies and, you know, bump into these colonial officials as a result. And just, you know, the interconnectedness, how we teach the British worlds or the British empire. The frequency of the days people are moving. The frequency in the distance, like spending 10 years in Singapore and then five years in the Caribbean and some time in New Brunswick and then Sierra Leone. Like it's kind of insane when you think about it. You'd think that building up like some cultural expertise in one part of the world would make sense to have that person stay in that part of the world. in Jamaica after maybe learning some of the language in Southeast Asia. Why would we double down on that experience? Let's just go send them to a completely different part of the world. So I think that gives us insight into the mentality of the Colonial Office that

[51:22] cultural awareness was not a priority. White supremacy was sort of just baked into the whole project that they were there to govern. They didn't have to know anything about the locals. so yeah i i learned quite a bit by the time they develop that local knowledge then they're scooped off to somewhere else yeah which they yeah are in a totally different context yeah presumption of the u.s fan rate services you don't want anyone to stay in one place too long because they might develop loyalties to the place instead of the country they're representing i expect it's the same idea yeah imagine it's the same idea built with solomon could have done a great opera and somebody even only posted places called victoria yeah some more observations on this spanish empire in the early modern period and i wonder you you You weren't ever able to determine family

[52:24] relationships, are you? Only if they're in Wikidata. So again, this is what's exciting, right? Right now, no. But we're increasingly going to have every census as open data. Like right now, the microdata mostly belongs to Ancestry. And so the microdata exists, but you have to sign contracts to use it. It'd be kind of a pain. but you can now get an LLM as Mark Humphries has shown us to read the census. So we're, we're planning to do that in Canada to just create a new copy that Ancestry doesn't own of the entire census. You know, are the models there this year? No, but are they going to be there by next year? Probably. And then we could try and find these people in the census and figure out who their dads were and who their brothers were. And that gives us a whole new angle on social mobility and empire.

[53:27] Yeah. Yeah, so for the really elite people, if they're in Wikidata and if the Wikipedia article has noted that family connection, but it would be a pretty limited thing for the top 1% at this point. You know, the aristocrats are another group that you can do it for their bloodlines have been recorded by weird people on the internet um and so you might be able to start looking at you know the gentry and the aristocrats in this service which would be interesting in and of itself because they yeah they have very interesting careers because it's still such a class-based uh political system that they still get sent out to work but they're clearly being chosen to be the future leaders so they are very different for you know they're not clerk three or whatever when they start their career at 20 they're put in as a adjunct to the the governor or something like that to learn the ropes of being a colonial official

[54:31] so yeah there might be ways just by looking for people with titles and trying to map that out yeah um i work in international health and historians are clearly way laid ahead of us in terms of use of ai for research i do qualitative research so i think it's like what we would do with historical texts we would do with documents or interviews and that sort of thing. I'm curious if you have any suggestions about where would be a good place to start if I wanted to do more of this kind of research or analysis? Because again, you're light years out of where we are right now. Yeah, but I started using Claude Code in May 2025.

[55:34] And yes, I'd done some text mining stuff before, but almost exclusively in groups where somebody else did all the technical work. And I was the annoying historian who kept on pointing out that it didn't work as well as they told me it was working. So that's the power of these tools. It can teach you as you go. I didn't really know what a knowledge graph was. and you know describing what i was trying to do claude said why are we using a relational database you should be using a knowledge graph okay what on earth is a knowledge graph it tells you then you go watch a few youtube videos you're like okay yeah knowledge graph's definitely a better tool for the kind of work that i do and you know all of a sudden you're realizing that grounding your knowledge graph to Wikidata helps you solve the disambiguation problem and then you become in three weeks an expert on Wikidata and figure out what an MCP server is you just sit and learn with these tools with your questions

[56:37] and it will help you figure out which of these methods that you know somebody in biology has already figured out or an enterprise a lot of the stuff i'm using is developed for enterprise like microsoft has tens of thousands of pdfs that they've created and they want to be able to interact with it through local AI agents. And they want answers that are highly accurate. They don't want BS injected into it. So they're the ones that formulate this idea of GraphRAG, where you build a knowledge graph at the heart of your retrieval augmentment generation system. And I've just sort of borrowed from these enterprise projects and figured out how to do it with open tools and get accurate results. But again, I'm pretty sure I didn't find the word GraphRAG on my own. Claude would have told me, oh, you should look into GraphRAG. What the hell is that? And so it's, yeah, if you're using it to learn, you'll learn

[57:38] way, way faster by just ponying up the hundred bucks a month, which is the uncomfortable part of it, right? As you're handing money off to these California tech bros that none of us like very much. But they're kind of a weird cult on top of that. Yeah. I just wondered, we circled out to the questions that you started to talk with, the questions you still haven't answered. Like, the difference between people who moved around and people who were pretty much spaining, how might we know that you have a fairly good media set? Could you vary using this level theory or something like, you know, people who didn't move and what attributes those people have? So I think it's pretty class-based or high flyers versus middling. So mostly if you're climbing towards being the governor, there's a high chance that you've worked in eight different colonies.

[58:40] If you're climbing towards being a regional police commissioner, there's a much better chance. And then if you're part of the local population that's been brought into the service, if you're a person from the Gold Coast, you're going to stay in the Gold Coast almost for sure, unless you're of European descent. So again, I'm not really a stats historian, but I edit a journal where people keep sending me all kinds of stats. I'm learning stats vicariously through historical methods. And then I say to Claude, so I think there's a journal article that I need to find a co-author. If anybody's interested, writing about imperial careers and know stats better than I do. I think there's a clean enough data set, but it's like the seventh paper on my list of papers I need to finish. I've got a sabbatical next year, but I've also been saying that we shouldn't all start writing seven papers a

[59:43] So I think that's about time, but yeah, quickly. So I bounced around because I don't have any love for that particular company. I think they're super problematic and cultish, but they're the ones that came out with Claude Code. So they're the first one that allowed you to work in the command line in folders on your computer where you're giving it access to your data and thinking with your data and now my university signed a deal with them and I'm getting it for free currently so and I don't know if any of the other ones are better I think we can just say none of them are good in terms of companies or at least if there's a good one their technology isn't very good like Mistral maybe in the eu is the one that seems the least bad but yeah their technology is way behind right now and not something i

[1:00:45] could probably use so we're kind of stuck between OpenAI meta now has a competitive product so if anybody's a big zuckerberg fan you can leave anthropic for meta but yeah it just seems like it's a bunch of bad options in terms of the corporations at this point. All right. Thanks so much for coming.