AgenticEvolve lets agents test ideas during program search — Persistent experiment agents — What does it mean?

The news. On October 22, 2026, Peng and colleagues posted AgenticEvolve to arXiv. Each evolutionary island is run by a persistent LLM agent that can run its own analysis scripts between formal evaluations. On online bin packing and a job-shop scheduling task it beat both a no-experiment control and OpenEvolve, and on twelve equation-discovery problems it recovered more ground-truth equations than the baselines. Read the paper →

Picture a restaurant group that improves its menu by evolution. Each round, a chef takes the best recipes so far, writes a new one from memory in a single sitting, and mails it to a critic. The critic tastes it, scores it, and never explains why. That is the standard recipe for LLM-driven evolutionary program search: the model sees parent programs and their scores, returns one candidate, and the evaluator scores it. The model never executes anything while it writes, so its only evidence about its own idea is the critic's score. Each tasting costs a unit of the budget, and the critic's test set is usually hidden.

AgenticEvolve keeps the evolutionary skeleton and changes the chef. Each island is run by one agent that stays in a single running conversation for the whole search. It edits program.py with read, write and edit tools, and it can write a second file, analysis.py, that imports the current program and runs it. The printed output comes back into the conversation, and the agent decides what to do next. Only the commit tool sends the program to the critic. Commit feedback is deliberately thin: metrics on success, only an exception type and location on failure, so a program cannot smuggle hidden data out through printed text. When the conversation nears the context limit, the agent writes a summary of what it tried, what it learned and its plan, and the conversation continues from that summary plus the current program.

Why would test batches matter if the critic's set is hidden? In the heuristic tasks the task text states how the hidden instances are generated, so the agent can generate its own instances from that distribution and measure behaviour on them. Three islands run at once, arranged in a ring. After a fixed number of commits, an island is shown its neighbours' best programs in the next archive summary and decides whether to read one and build on it. This is also a reasoning-budget choice: tokens spent on experiments are only worth it if they replace blind guesses. The paper tests that choice directly by comparing against No-analysis, which removes only run_analysis and keeps everything else identical.

Hold the setup fixed: online bin packing, DeepSeek-V4.1-Flash, three islands, and five runs per method. Excess is the percentage of extra bins over a lower bound, so lower is better. No-analysis lands at 5.54% and AgenticEvolve at 3.42%, a 2.12 percentage-point difference in mean excess in this five-run comparison. The authors also traced one commit in a GLM run, where the agent ran ten analyses before submitting. It generated 400 sequences and found that bins left with less than 20 units, which no item can fill, held 58% of the total waste. It then fitted a table of expected leftover waste at 13 points, turned it into a placement rule, and tested it locally. On the hidden set, that first commit cut the excess from 6.553% to 5.392%. That local analysis found a pattern the aggregate score alone did not show.

The second regime reverses the setup: in equation discovery the training data is available, so the agent can fit candidate formulas directly. On twelve problems from LLM-SRBench, with one run per problem, AgenticEvolve met the out-of-domain accuracy tolerance on all twelve, against eight for No-analysis, eight for SR-Scientist, three for LLM-SR and one for PySR. It also made 316 evaluator calls in total, versus 3,988 for No-analysis, which fits the idea that the agent checks formulas locally before committing. The estimated cost was $6.54, against $12.12 for No-analysis and $111.34 for LLM-SR.

Read the result with its limits in view. This is a preprint from one group, with five runs per method on the heuristic tasks and a single run per problem for equations. The experiments are not free: the cost advantage depends on the model and the budget. With GLM and Qwen, repeated cached input dominated AgenticEvolve's cost (about 87% and 78%), and OpenEvolve was cheaper there. On the job-shop task, No-analysis made cheap early progress before AgenticEvolve caught up. The paper also notes that its execution uses access restrictions rather than an operating-system sandbox. The durable lesson is that an agent inside a search loop earns its cost when it can measure its own idea before spending a scarce, hidden evaluation. This echoes the chef's back kitchen: the critic is still the judge, but the cooking happens before the tasting. Context compaction like the agent's own summaries is one of the standard fixes for long conversations.

Goes deeper in: AI Agents → Planning & Reflection → When to Pause and Observe