Assessing the usefulness of ChatGPT in crafting interpretive content

02 November 2023 – Catherine Stiers

Shortly after ChatGPT was released to the public in late 2022, the debate started about whether or not it meant the end of public history as we know it. Is the public historian obsolete when computers can tell you anything you want to know about the past? While many classrooms around the country have outright banned ChatGPT, some professors are using AI-generated content to uncover what the tool can and cannot do for the field of historical research. Part of my job is to create social media content using archival materials, which is a worthwhile yet time-consuming task. I wanted to know if ChatGPT could help me, so I set out to test the viability of using ChatGPT as a tool to create history-based content more expeditiously.

The basic version of ChatGPT is free and easy to use, making it a tempting tool in the world of public history-related content creation. The interface is similar to that of a messaging app, but the person on the other side is a language model-based chatbot rather than another human being. The barrier to entry is low; simply type in what you want created and ChatGPT will make it happen.

Creating social media content is an infamously fast-paced task, even when the subjects are historical. It can be a challenge to create content that is relevant, engaging, and timely, particularly when online trends change seemingly by the hour. Many people use ChatGPT for drafting emails, creating promotional messages for events, etc., but I wanted to know if ChatGPT was up to the challenge of interpreting a complicated piece of history.

For this experiment, I used archival material I wrote interpretive text for as part of my job as the Research and Outreach Specialist at the College of Charleston, an 1849 slave pass issued to an enslaved woman named Grace.

A pass for an enslaved woman named Grace. The pass reads: “Grace has my permission to sleep In Patton’s lot for three months from Oct. 3, 1849,” and is signed by Isaac L. Ridell, 70 Meeting Street. The words “Found in Book of Common Prayer presented by Daniel Horlbeck 1934” are written lightly in pencil. Image credit: Lowcountry Digital Library, College of Charleston Libraries

I wanted to convey several points when writing social media interpretation because a substantial amount of backstory is required to fully understand the context of this simple slip of paper. Firstly, I wanted Grace’s name to be emphasized, and to center her experience rather than that of her enslaver. Secondly, I wanted to make it clear that urban enslaved people did not deal with the same environment as enslaved people on rural plantations. I knew most casual readers of our Instagram account would only be familiar with the plantation model of enslavement and would not know that urban Charleston was a unique environment. Thirdly, I wanted to make it clear why the pass was so important; in this case, the interpretation needed to emphasize why Grace would be in danger without it.

I wrote the following brief interpretation for an Instagram post without the influence of AI. (View the transcripts of screenshots via this Word doc or this pdf):

Screenshot of Instagram post written by author without consulting ChatGPT.

I then prompted the free version of ChatGPT:

Screenshot of first ChatGPT result.

For the first prompt attempt, I purposefully did not give ChatGPT very much information to work with. ChatGPT cannot “see” the image; it relies on our descriptions. The results are predictably vague. While the suggested interpretation does center the experience of the enslaved person, it fails to provide any tangible information on the history of enslavement in Charleston. You could replace the word “Charleston” with any other city in the South, and the text would remain the same. ChatGPT also has a bad habit of employing empty platitudes as exemplified by the hashtag #NeverForget, more commonly used to remember 9/11 or the Holocaust. This suggested interpretation is fine and relatively inoffensive, but it doesn’t have very much to say. What shouldn’t we forget about the past? What were the hardships referred to? What does this slip of paper have to do with strength or resilience?

Screenshot of second ChatGPT query.

Screenshot of second ChatGPT result.

I provided more information for my second attempt. This time, I gave ChatGPT the text written on the document as well as three points I wanted to make clear in the response. In some ways, the results were what I asked for. The response does have Grace’s name front and center, mentioning her in the title and throughout the text. Again, the language is cliched (“sheds light,” “strive for a future,” etc.) and perpetuates false stereotypes about archives being dingy and unorganized basements where historical documents can be “stumbled upon” rather than institutions run by professional archivists whose job it is to arrange, describe, and make available for research every slip of paper in their collections.

Most egregiously, ChatGPT clearly misunderstands what a slave pass is, as evidenced by the quote, “In a society where freedom was a distant dream, this pass was her lifeline, shielding her from vulnerability and providing a glimpse of hope.” There was nothing hopeful about this document. Slave passes were a form of surveillance, giving white patrollers an excuse to stop and interrogate any Black person they saw moving about the city. Some enslavers hired enslaved laborers out to work in other households for profit. Grace also could have been given permission to visit her friends or family living downtown. We don’t know why Isaac Riddell issued the pass to Grace, but it almost certainly did not “[provide] a glimpse of hope.”

Screenshot of third ChatGPT query.

Screenshot of third ChatGPT result.

While still full of vague phrases and empty platitudes, this response offers a few promising tidbits. This time, I gave ChatGPT specific instructions regarding tone and asked it not to include emojis (a request that was apparently denied or ignored). It also included an idea in its response not explicitly provided by my prompt, that city life for enslaved people meant an increase in surveillance.

The response also somewhat describes what slave patrols were, “armed groups . . . tasked with maintaining the status quo and ensuring that enslaved individuals remained subjugated.” However, the description never tells the reader who slave patrols were composed of and what the “status quo” was. Most interestingly, it alludes to a connection between the legacy of slave patrols and the “fabric of our society,” but fails to explicitly connect the practice of slave patrols with modern policing. Although users may come to expect a certain degree of neutrality from AI, the information is subject to algorithms that sort through massive amounts of text online. The text sources themselves are only as neutral as the human who wrote them, making the results of ChatGPT unpredictable. The third attempt was the most promising, but it required extensive reworking of the prompt. At this point, the prompt was so specific that I might as well have written the post myself.

In my experience, using ChatGPT to create interpretive social media content is not the shortcut we might hope for. However, there is more room for experimentation. The Nasher Museum of Art at Duke University recently opened an exhibit written by AI, Act as if you are a Curator: an AI-generated Exhibition. The exhibit was organized by ChatGPT, but it required a team of humans to develop prompts and physically install the materials. Even so, this will almost certainly not be the last exhibit curated by artificial intelligence.

In my experience trying out different prompts, ChatGPT was most successful at writing a brief, fluffy social media post for Mother’s Day to accompany a cute picture of a mother sea lion and her pup.

Mother sea lion with cub, 1962. Image credit: John Henry Dick, John Henry Dick Collection, Lowcountry Digital Library, College of Charleston Libraries

Screenshot of Mother’s Day ChatGPT result

Sure, it’s a little cheesy and vague, but that works when the goal of the post is to use social media to showcase something interesting and timely from the collection rather than to teach something more serious.

ChatGPT works best when it is given specific prompts including guidelines for tone, but it is not always successful at interpreting what the user wants. If GLAM institutions decide to use ChatGPT as a tool for social media creation, they should be careful not to make more work for themselves through the extensive edits necessary to interpret history as well as a trained human professional. I plan to continue using ChatGPT for inspiration, but I would not trust it to interpret anything beyond surface-level content.

Note: As of late September 2023, ChatGPT is now capable of analyzing images.

This experiment was conducted using the Default (GPT-3.5) model.

~Catherine Stiers is a public historian currently located in Charleston, South Carolina. You can find her (and her little dog, Charlotte) on Twitter or LinkedIn.