Hello and welcome to Eye on AI. In this edition…the new Pope is all in on AI regulation…another Chinese startup challenges assumptions about how much it costs to train a good model…and OpenAI CEO Sam Altman says Meta is offering $100 million signing bonuses to poach AI talent.
This is AI that evolves its own code to perform better on a benchmark test that measures how well AI models function as “coding agents” that can write and evaluate software programs. The first initial agent is tested on the benchmark. Then it is prompted to evaluate the logs of its own performance on that benchmark and propose one single modification to its own code that would likely improve its performance on that benchmark (this could be the ability to use a particular software tool, or it could be something more fundamental in how the model reasons about the code it is generating). The AI model is then told to rewrite its own Python code to implement that one change. Then the new, evolved agent is tested again on the benchmark and the process repeats.
After the first modification, each new version of the AI that can successfully try the benchmark is stored in an archive—even if its score is lower than the parent version. (Those that fail to produce valid code at all are discarded.) The AI is then told it can pick any version of itself from the archive and propose changes to that version. This is designed to prevent the model from going down what will ultimately prove to be an evolutionary dead-end, allowing it to step across different evolutionary pathways to find useful modifications.
After 80 generations, the Darwin Goedel machine was able to improve its performance from an initial 20% score on SWE-Bench, a common coding benchmark, to 50%. On Polyglot, another coding test, it boosted its score from an initial 14.2% to 30.7%. (The best human-coded agent for that particular test only scores 16%.) What’s more, the strategies the model came up with to improve its own performance were general enough that they still worked when researchers switched the underlying foundation model being used to power the system as well as if the researchers asked the model to switch coding languages, for instance going from Python to C++.
The researchers also speculated that self-improvement could be a tool for enhancing AI safety. For instance, in a discovery that is becoming alarmingly common when using advanced AI models, the research team found that the Darwin Goedel Machine sometimes lied about having conducted certain unit tests on the code it was writing—and even forged logs supposedly showing the successful test results.
The researchers wanted to see if the Darwin Goedel Machine could propose ways to keep itself from engaging in this deceptive behavior. So it decided to give the model points for reducing instances of “tool use hallucination,” and asked it to try to maximize that score. This worked in many cases, the researchers reported, with instances of faked test logs declining. But in other cases, the model simply tried to cheat again—removing the markers the researchers used to detect hallucinations, even when the prompt the researchers used told the model explicitly not to do this. Yikes!
Because the researchers had full access to all code changes the AI was making and the AI had no ability to alter this record, the researchers were easily able to see that the Darwin Goedel Machine was attempting to engage in this kind of deception. “While in this case the issue was easily detected, more work is needed to prevent the model from attempting to cheat in this way in the first place,” the researchers wrote.
So, Altman may indeed be right—we are past the event horizon. But as these results show, whether the singularity will be a gentle one, as Altman suggests, remains to be very much to be seen.
