In the first article in this series about measuring AI creativity, I let 8 AI models play a game of AI Categories (see github repo here), loosely inspired by Scattergories. Claude Sonnet was a head-and-shoulders champion, while DeepSeek and GPT-4 held their own near the top.
In Part 2, the AIs got better results when shown what their reasoning and response had been on the prior round, and even better results still once they were shown all other AIs’ responses (without reasoning, if any) from the prior round.
This additional information improved Claude Haiku from one of the lowest-performing models in the competition, to a point or two shy of the top 3 performers. Meanwhile, the champion Claude Sonnet took advantage of the same information to develop innovative strategies for exploiting the game rules to score more points, causing it to output more interesting and unexpected content in the process.
Researchers have found that transformer-based models can emulate learning algorithms during model inference without adjusting model weights. (What learning algorithm is in-context learning? Investigations with linear models, Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, Denny Zhou.) Qualitatively, this seems like a possible example of Sonnet employing emergent online reinforcement learning through the game-imposed feedback loop of inferring and reinferring using previous successful outputs as input.
Claude models show the most creative adaptation
It stands out that the Anthropic Claude models, in particular, showed the greatest online adaptation to in-game incentives. Other models, like the GPT family, DeepSeek, and Mistral, while they may have performed quite well on the test itself, displayed a much narrower and more predictable band of behavior, and their output was mostly unaffected by additional feedback about game performance.
This seems notable in any discussion of creativity. We see creativity and innovation arising from iterated interaction with a constrained environment that gives some form of feedback. This feels intuitively correct! From a measurement point of view, it’s probably not enough to look at an AI’s ranking under a single condition. It might be more informative to look at how its score and/or ranking change when moving from a more restricted condition to one with opportunities for feedback loops and social learning.
Let’s look at how scores changed for all models between the Blind and Open game conditions:
A few things are evident from the chart. First, every model performed better when responses to the previous round were displayed. It’s not totally clear whether every single model benefitted directly from the open conditions, or whether the general improvement is merely an example of the “rising tide lifts all boats” property of the Categories game, where the overall performance of the group tends to improve as the performance of any of the individuals improve, since the only way to score is to provide unique answers, thereby reducing duplication of other players’ responses as well. This question is answered in future posts.
A second observation is the diminishing returns as we move from lower-performing models to higher-performing ones. This is unsurprising in many ways. The lower-performing models, of course, have more room for potential improvement, and the higher-performing models have fewer remaining points possible to earn. In this game, for example, the maximum score is 50, and Claude Sonnet already scored 39 under the Blind condition.
A better measure of creative adaptation?
That means it’s only partially informative to talk about raw points gained, or even percent improvement in score. Another way to measure this improvement is to think about the number of possible additional points and what percent of those points the model was able to gain by providing opportunities for adaptation. Let’s call that the Adaptation Score. By definition, this score has a possible maximum of 100% for all models, so we can rank models by how close they approach this maximum under increasingly open information conditions.
If, for example, an AI scores 45 of 50 points in the Blind condition, that means there’s only 5 more points possible. If the AI then scores 48 of 50 points in the Open condition, that means it has improved by 3 out of 5 possible additional points, or 60%.
While in contrast, consider an AI that first scored 10 out of 50 points, leaving 40 points possible, and then improved to 20 points. This would be an improvement of 10 out of 40 possible additional points, or 25%.
The following chart shows adaptive improvement scores for all models in the group.
It’s reassuring that this metric reflects exactly what was qualitatively evident in the data: the Claude models showed greatest adaptation to the game conditions, while the other models showed more rigid responses. Furthermore, the metric is sufficiently calibrated to demonstrate the strength of improvement both for Claude Haiku, which made a large absolute improvement over an initially poor performance, as well as Claude Sonnet, which made a small absolute improvement over an already high performance.