And one of the sites that we have been following is a computational biology lab, which is developing a benchmark for coding agents. So what the team does is that it turns pieces of ordinary lab work into repeatable tasks, which allows them to compare agents under different testing conditions.
They're asking whether an agent can complete a well-scoped research task in a way that a scientist would recognize as methodologically acceptable. So in this task, the scientists were trying to map how fruit flies respond to light by tracking changes in their movement.
The agent was asked to compare the flies' forward speeds when the light was on and when it was off. The appropriate method, in this case, would be to calculate the on and off averages for each fly and then average them across the group.
So what the agent did as an output was to produce a very tidy file, and its final values fell within the benchmark's tolerance. So at first, everything looked fine, and it seemed like the agent was successful in the task