And one of the sites that we have been following is a computational biology lab, which is developing a benchmark for coding agents. So what the team does is that it turns pieces of ordinary lab work into repeatable tasks, which allows them to compare agents under different testing conditions.
They're asking whether an agent can complete a well-scoped research task in a way that a scientist would recognize as methodologically acceptable. So in this task, the scientists were trying to map how fruit flies respond to light by tracking changes in their movement.
The agent was asked to compare the flies' forward speeds when the light was on and when it was off. The appropriate method, in this case, would be to calculate the on and off averages for each fly and then average them across the group.
So what the agent did as an output was to produce a very tidy file, and its final values fell within the benchmark's tolerance. So at first, everything looked fine, and it seemed like the agent was successful in the task.
But when the researchers examined how it had reached the answer, they found that it had skipped the per-fly calculation. So what the agent had done was to select one fly with a specially well-labeled trajectory and use that fly as a proxy for the whole group.
By chance, that fly was close to the true group average, so the answer passed the benchmark. So as one researcher put it to me later on, he said that the agent was numerically right but methodologically wrong.
The goal was appropriate, the output was successful, but what had gone wrong was the way the agent interpreted and carried out that task. It shortcut, erased the variation across flies that the scientific procedure was meant to preserve.
So this is the goal-plan-execution gap in practice. When we assign a goal to an agent, we also give it some discretion over how to accomplish that goal.
Its autonomy grows when it can carry that discretion into action. But that leaves us with this question of where to draw this boundary between what is useful delegation and what is an agent's autonomy.