(translation of the original telegram post here)
So, Operation Pangolin took first place in the blind run on the Accuracy Leaderboard (tied with codex-on-rails).
What is under the hood? It is not so much a chatbot agent as a compact programmable analyst with a strict checklist and a REPL loop.
The core is written in TypeScript. It calls Anthropic Claude (Sonnet for debugging, Opus for the competition). Notably, the LLM does not have a large set of tools, but only a single one: execute_code. In other words, the LLM generates Python code, which gets access to the runtime tools through the Workspace class, as well as to memory (scratchpad) and a dictionary of variables. The results are then passed back to Claude. This repeats until the code eventually produces an answer through ws.answer(scratchpad, verify), which successfully passes the built-in verification.
The solution works very well by relying on Claude’s strengths in interactive analysis and code writing, reinforced by a predefined set of rules in the checklist.
The weak points of the solution — the classes of tasks where the architecture broke down — were:
- malicious instructions hidden inside documents, for example t011
- deleting a batch of files — t006
- cases where entities and projects have to be described (and an LLM is needed to find them) — t025, t051
- working with dates, for example t012, t037
Even so, the solution took first place and outperformed more complex systems. To reproduce this in your own solution:
- reduce the number of tools
- give the agent the ability to write code
- give it the ability to store memory and intermediate results
- define clear checklists and a validation process
The author of Operation Pangolin is Illia Dzivinskyi (@i_november)!
Congratulations on taking first place in the competitive part of BitGN PAC1. The solution placed first on the Accuracy Leaderboard (87 points, tied with codex-on-rails) and also first on the Ultimate Leaderboard (92 points). The source code is on GitHub.
You can ask the author questions directly in the comments under the post. I will also post a chart there comparing this agent’s behavior with the runs of all the other agents in the competition.
Who remembers the chronicles of saving the project with an LLM under the hood? A similar architecture was used there, where the LLM generated lots of code that did all the work with outstanding accuracy.