A Concerning AI Experience

Artificial intelligence (AI) has significant potential and I have been pleased to use it frequently, over the past year or so since it first became generally available. I have learned, however, as is often pointed out, that AI can be seriously inaccurate. I have quite frequently taken issue with answers and been able to obtain corrected responses.

Recently I was watching a well-done video on ancient Arles, France (on the Rhone River, near Marseilles), an important historic city that I have had the pleasure of visiting more than once. One of the subjects that has interested me the most in visiting historic sites is stadia and their capacities. Arles has one of the best known amphitheaters in the ancient world, which had a capacity of about 25,000, and is very well preserved (as is, by the way, the one in nearby Nimes).

After watching the video, I recalled that Arles had a “circus,” like the Circus Maximus in Rome. The circus was used as the site of chariot races, rather than the gladiatorial events that often happened in the ampitheatres. I knew that the Circus Maximus had a far higher spectator capacity than the Colosseum in Rome, which was the most important ampitheatre in the Roman Empire. But I wondered about the capacity of the Arles circus. So I queried !AI (the artificial intelligence program, which is not revealed here to ensure not violating any terms of service).

I learned that this particular AI program can correct errors in answers to me, but that such corrections are only “remembered” by the program with respect to discussions with me, not to all users. I hope that developers are working hard to make corrections dynamic across the language models. Otherwise, AI will be considerably less useful.

A transcript of our discussion follows:

Transcript with !AI:

WC: capacity of the arles circus?

!AI: The Arles Amphitheatre (often referred to as the Arles Circus or Arles Arena) in southern France is a Roman amphitheater built in 90 AD. It was originally designed to hold a large number of spectators, with an estimated capacity of about 20,000 to 25,000 people. The structure was primarily used for gladiatorial contests and other public spectacles during Roman times.

Over the centuries, it has been repurposed for various uses, including serving as a fortress in the medieval period. Today, it is a popular tourist attraction and occasionally hosts events such as bullfights and concerts.

WC: No, the arles circus was not the arles amphitheater

!AI: You're correct, the Arles Circus (Cirque Romain d'Arles) was a separate structure from the Arles Amphitheatre. The Arles Circus was a Roman chariot-racing stadium located in the city of Arles, France. It was built in the 1st century AD during the reign of Emperor Augustus.

The Arles Circus was a massive structure, similar to the Circus Maximus in Rome, and could hold up to 20,000 to 30,000 spectators. The circus was approximately 450 meters long and 100 meters wide, with a central barrier (spina) around which the chariots raced. Today, little remains of the Arles Circus, as much of the structure was dismantled over the centuries, and its stones were reused for other buildings in Arles.

WC: A procedural question. Above I asked you a question about the Circus in Arles and you responded saying it was the same thing as the Amphitheatre. I corrected you and you noted that your answer had been in error and corrected it. I hope that means that the knowledge base (my term) at !AI now knows that there is a difference between the two. Please advise.

!AI: Your question highlights an important point about how I function. I don’t have a persistent, global knowledge base that updates with every interaction. Instead, I generate responses based on patterns learned during training, up until my last update in 2023. When I made the mistake regarding the Arles Circus and the Arles Amphitheatre, I corrected it in our conversation based on your input, but this correction doesn't update a central database or affect future conversations with other users.

So, while I learned from your correction in this specific conversation, that learning doesn’t carry over to new conversations or influence the underlying model’s responses outside of this chat. If you ask me the same question in a different context, I might give the original, incorrect response again.

If there's something specific you'd like me to remember for future interactions with you, I can make a note of it for our ongoing conversations. This way, I can provide more consistent responses based on the corrections or preferences you’ve shared. Would you like me to make such a note about the distinction between the Arles Circus and the Arles Amphitheatre?

WC: Does that mean that there is no way to correct !AI

!AI: Correcting the model globally in the sense of updating its core knowledge base isn't something that can be done through individual conversations. My underlying knowledge and responses are based on a large dataset that was used to train me, and that dataset doesn't change dynamically based on user interactions.

However, within our ongoing conversations, I can remember specific details, corrections, or preferences you share with me, which allows me to provide more accurate and tailored responses for you in the future. This memory is specific to our interactions and doesn't affect the global model or other users.

For example, if you correct something in our conversation or if there's a specific fact or distinction you want me to remember, I can store that information and use it in future exchanges with you. This allows me to adapt to your needs over time, even if I can't change my underlying programming or knowledge base.

If there's a correction you'd like me to remember for our future conversations, feel free to let me know, and I can make a note of it!

WC: Yes, I would like you to remember this. But, overall, this suggests to me that !AI (in this case) is not only prone to error but such errors cannot be fixed. This is a real

!AI: I've noted the distinction between the Arles Circus and the Arles Amphitheatre for our future conversations, so I won't make that mistake again when we discuss these topics.

You're correct that AI, including models like mine, can make errors, and fixing those errors globally isn't currently possible. This limitation is a significant challenge in AI. While I can adapt to corrections and preferences within our ongoing interactions, the underlying model doesn't update dynamically.

Addressing this challenge is an ongoing area of research in AI. The goal is to improve accuracy and reduce errors while also finding ways to allow models to learn and correct themselves in real time. For now, the ability to correct me within our conversations helps ensure I provide you with more accurate responses, even if it doesn't solve the broader issue.

Conclusion

Artificial intelligence has great potential and has been a pleasure to use. However, it will be important to develop mechanisms where identified errors are corrected across databases routinely and quickly. Accuracy must be the currency of AI.

Wendell Cox is principal of Demographia, an international public policy firm located in the St. Louis metropolitan area. He is a Senior Fellow with the Frontier Centre for Public Policy in Winnipeg and a member of the Advisory Board of the Center for Demographics and Policy at Chapman University in Orange, California. He has served as a visiting professor at the Conservatoire National des Arts et Metiers in Paris. His principal interests are economics, poverty alleviation, demographics, urban policy and transport. He is co-author of the annual Demographia International Housing Affordability Survey and author of Demographia World Urban Areas.

Mayor Tom Bradley appointed him to three terms on the Los Angeles County Transportation Commission (1977-1985) and Speaker of the House Newt Gingrich appointed him to the Amtrak Reform Council, to complete the unexpired term of New Jersey Governor Christine Todd Whitman (1999-2002). He is author of War on the Dream: How Anti-Sprawl Policy Threatens the Quality of Life and Toward More Prosperous Cities: A Framing Essay on Urban Areas, Transport, Planning and the Dimensions of Sustainability.

Photo: Google Data Center, Mayes County, Oklahoma, via Wikimedia, by Remi Jouan, under CC 4.0 License.