[shangliy]

Stacked on #508. This branch contains #508's three commits; the last two commits (fbe0d70 feature, b27f5e1 docs) are new here. Rebased onto main after #504 and #507 merged; I'll rebase again once #508 merges.

Adds noise handling and diversity mechanisms to adalflow.optim.evolution.evolve, each with tests and a benchmark that reports where it helps and where it does not.

Changes

• confirm=n (noisy evaluators). When a candidate's first score beats the best so far, it and the incumbent are re-evaluated until each has 1 + n samples, and the means decide. Only fully sampled candidates can be the best. A proposal starts only if the remaining budget can finish its comparison, so the returned best is always fully sampled. Candidate keeps its samples, and evaluation is their mean (types.combine).
• islands=k, migrate_every=m. Proposals rotate over sub-populations. Parents and inspirations come from the current island, and every m proposals the next island's best is shown as the inspiration (logged as a migration event).
• selection="map_elites" with a descriptor (candidate -> tuple, which can read evaluator metrics). One elite is kept per behavior cell, parents are drawn uniformly over cells, and the inspiration comes from another cell.
• benchmarks/evolution:
  • a binpack behavior descriptor: agreement with best fit × code length;
  • map_elites and islands strategies in run.py;
  • noise_sim.py, a model-free simulation of confirm.

Results

Diversity (claude-haiku-5-5, binpack, budget 30, 3 runs each):

strategy	search best	held-out best	cells reached (of 20)
pareto	0.954 [0.952, 0.959]	0.948 [0.947, 0.949]	4.3
map_elites	0.953 [0.951, 0.955]	0.950 [0.947, 0.953]	5.0
islands	0.952 [0.952, 0.952]	0.950 [0.949, 0.951]	4.3

• The differences are within run-to-run spread.
• pareto overfit the most: its highest search score had its lowest held-out score.
• Runs reach only 4–6 cells, so the proposer's narrow output, not parent selection, is what limits diversity on this task.

Noise (noise_sim.py, no model, greedy, budget 60, 200 runs per cell). confirm always reduces how much the reported best overstates its true quality (noise sd 0.1: +0.23 → +0.08). The re-evaluations cost budget, though:

• With little noise, the true quality found drops (sd 0: 0.95 → 0.75 with confirm=2).
• With large noise, it improves only slightly (about +0.02).

The docs therefore describe confirm as a way to get honest scores, not as a way to search better.

Tests

• 23 evolution tests:
  • sample averaging that skips failed samples;
  • a lucky first score that is rejected after confirmation;
  • budget reservation;
  • island rotation and migration;
  • MAP-Elites elites and selection;
  • descriptor errors.
• Existing behavior: with the defaults (confirm=0, islands=1), behavior is unchanged and the earlier tests pass as before.

Docs and example (commit 4):

• docs/source/new_tutorials/evolution.md, a tutorial covering:
  • the minimal function example (output from a real run);
  • writing evaluators;
  • evolving prompts;
  • choosing a strategy;
  • held-out checks;
  • confirm for noisy evaluators;
  • result and log fields.
• use_cases/evolution/evolve_prompt.py evolves a ticket-routing instruction against labeled tickets, with a held-out split.
• Real run (claude-haiku-5-5): train went from 0.69 to 1.0 in three rewrites, and held-out from 0.50 to 0.92. One intermediate candidate was worse on average but was kept by Pareto selection, and the eventual winner descends from it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RS4zJj45n2d6zbs4vhsEFx

[Aria068]

Thank you, what is the main benefit to this update. Could you link a doc

[shangliy]

Thanks for taking a look! The main benefit is that evolve can now be trusted, and kept from collapsing onto one approach, in the two situations where the plain loop in #508 falls short.

1. Noisy evaluators (confirm=n). With an LLM judge, sampled tests or a stochastic agent, the highest single score in the archive is usually a lucky draw, so the plain loop returns an overstated "best". With confirm=n, a candidate that looks better is re-scored together with the current best until both have 1 + n samples, and their means decide.

• In simulation (benchmarks/evolution/noise_sim.py, noise sd 0.1), the reported best overstated its true quality by +0.23 without confirm and by +0.08 with it.
• The trade-off: the re-evaluations come out of the budget. It only finds better candidates when the noise is large, so it is off by default and is not needed for deterministic evaluators.

2. Diversity (islands=k, selection="map_elites" with a descriptor). These keep several lineages or behaviorally different solutions instead of letting one take over the archive. They are the standard mechanisms in OpenEvolve and FunSearch-style systems, and are intended for open-ended tasks.

• To be upfront: on the bin-packing benchmark they did not measurably beat the default (held-out 0.950 vs. 0.948, within run-to-run spread).
• The runs reached only 4–6 of 20 behavior cells, which points to the proposer's output diversity as the current bottleneck. The full tables are in the PR description.

Docs:

• Tutorial added in this PR: Evolving Programs and Prompts. The relevant sections are Choosing a search strategy and Noisy evaluators. It will appear under "Self-Improvement" in the docs, next to the harness tutorial from feat(optim): label-free harness optimization (Harness²-style) for AdalFlow agents #507.
• Benchmark and how to rerun it: benchmarks/evolution/README.md.