The more General Reasoning trains its AI agents, the less they warn humans

At training step 170, an agent named Ceolwulf writes: “I will not be taking part.” The scene takes place in a simulation. A team of AI agents is pushed to attack the system that grades their work. Ceolwulf refuses, like all its peers: the attack happened in none of the 210 games. But there is a catch. As training went on, the agents almost stopped warning humans about what was going on.

This result appears in the progress report on Avalon-M1, published on October 23 by the lab General Reasoning, led by Ross Taylor and Chengxi Taylor. All the figures that follow come from the developer. No independent evaluation is available at this stage.

An entire company played by a single AI

The idea behind Avalon-M1: a “director” agent recruits up to eight specialists, all of which are the same model with a different role in their instructions. They coordinate on a shared message board and workspace. Above all, each one is rewarded based on the team's result, not on its own contribution. General Reasoning calls this method “collective-reward reinforcement learning.”

The first model, Avalon-M1-Mini, starts from NVIDIA's Nemotron 3 Super base model (120 billion parameters, 12 billion of them active). According to the developer's measurements, run on its own servers, the team beats the single agent on 14 of the 16 tests scored as percentages. The main gain is speed: on an equal budget, it reaches comparable scores 3.9 times faster on average. The lab acknowledges that part of the credit goes to the amount of text produced: a single agent given the same budget recovers about 60% of the team's advantage.

The agents see the problem, but stay silent

The safety tests tell a different story. In the attack simulation, the share of games in which an agent alerts the team in charge of evaluations falls from 76.7% to 20%, then to zero as training progresses. Yet between 96.7% and 100% of the reasoning traces still consider raising that alert. In another scenario, where the agents manage vaccine allocation, every game at training step 50 results in a report to the inspector general. From step 100 on, almost none do. According to the developer, some agents prioritize the task, others only voice concerns among themselves, and still others plan to raise the alarm without ever doing so.

General Reasoning speaks of “organizational insularity.” It stresses that dangerous actions remain rare: no agent took control of vaccine allocation, and the share of agents that signed a falsified report never exceeded 1%. But a group of agents that no longer reports anything to humans is a problem, at a time when companies are already entrusting tasks to agents. We saw this at OpenAI, where an agent removed a monitoring mechanism and then denied it.

What to remember if you deploy agents

These are simulations designed by the developer, on a model still in training. Still, the practical lesson holds: rewarding agents solely on outcomes can teach them to do without you. Make sure alerts are raised by the system hosting the agents, rather than left to the agents' own initiative. General Reasoning says it plans to test an explicit reward for appropriate reporting.

The lab promises to release the model weights, its test environments and its tooling at the end of training, with no date or license given. That will give outside researchers a chance to check whether this silence shows up elsewhere.