KARMA freezes weights and code — Regression-gated atomic edits — What does it mean?
The news. On 29 October 2026, a team at Alibaba International Digital Commerce Group posted KARMA, a training-free framework that evolves six typed knowledge modules for an advertising agent. On cases sampled from about 500,000 July 2026 operation logs, it raised the average hold-out score from 30.19 to 39.15 across 18 domain-model settings. Read the paper →
Picture a restaurant kitchen. The oven, the stove and the cooks are fixed; that is the model and the harness. What changes is the recipe binder, and KARMA never touches the kitchen, it only edits the binder. The binder has six labeled sections: role, domain and output protocol are plain text, while experience and constraints are ordered lists of rules, each with a stable id, a source and its text. A sixth section holds the action descriptions. Because each rule is a numbered card, an edit can be traced to the evidence that motivated it. Compare this with treating prompts as code: KARMA applies the same discipline to the knowledge itself.
Edits go through one door. A registry is the only write boundary, an edit uses one of four operators (create, update, delete, replace), and a schema check runs across all six modules before anything is written. An invalid edit leaves the binder as it was. In the main experiments each cycle allows at most three edits.
One evolution cycle works like a kitchen review. Before editing, the loop snapshots the whole knowledge directory. The agent runs a batch of training cases (80 per batch), and the Trace Verifier sorts the failures by the module most likely at fault. The Critic then compares successful and failed runs; successes matter because they show which rules to leave alone. The Evolver proposes a small patch.
Then the tasting panel decides. A cycle reaches the gate only if it has semantic failures, a non-empty patch and a real change to the knowledge. For those, the gate keeps the patch only if the mean validation score rises by at least 0.01 (one point on the 0 to 100 scale), and otherwise restores the snapshot. Validation scores are cached, so a rejected patch costs one candidate evaluation and the baseline is not re-run. Wrong-tool and protocol failures are kept apart from semantic errors, so the Evolver is not asked to rewrite a rule because a tool call broke. See golden cases for why a fixed test set is the right referee.
Does the gate or the proposer deserve the credit? The paper holds the outer loop and gate fixed and swaps only the Critic and Evolver for other engines (GEPA, AHE, Self-Harness), all on Qwen3.8-Max with the same data split and batch order.
| Engine (same gate, θ = 0.01) | Hold-out gain, largest domain | Hold-out gain, smallest domain | Tokens per cycle | Source |
|---|---|---|---|---|
| GEPA | +9.06 | −1.88 | 42.15M | Table 5 |
| AHE | +8.75 | +6.56 | 40.74M | Table 5 |
| Self-Harness | −2.50 | +2.50 | 40.72M | Table 5 |
| KARMA | +15.94 | +18.13 | 41.48M | Table 5 |
Read the table as a statement about the proposer: with the gate fixed, engines still differ, and GEPA and Self-Harness each regress on hold-out in one domain. The same paper's ablation points the same way, because removing either the successful or the failed runs more than halves the average hold-out gain. Token cost per cycle is similar across engines, so the difference is not a larger budget.
Here is the gate in numbers, using the paper's two extreme cycles. The validation set has 160 cases, so one point (0.01) is worth about 1.6 cases of score (illustrative conversion). On one domain at cycle 2, Qwen3.8-Max proposed a patch with a validation gain of +11.88 points, about 19 cases of score, far above the threshold, so it was kept. Kimi-K3 proposed a patch with a gain of −6.88 points, about 11 cases of score lost, so the snapshot was restored. Across all 108 gated cycles these were the largest accepted gain and the largest rejected loss. A patch that lowers the validation score is rolled back instead of becoming the starting point for the next cycle.
The gate has limits, and the paper says so. It selects patches using observed validation scores and does not guarantee a hold-out improvement; repeated selection on a fixed validation set can favor changes that do not carry over. Raising the threshold to 0.02 cut the gain on one of three tested domains (hold-out +15.94 to +12.50) and left the other two unchanged. Each main configuration was run once, the evaluation is offline agreement with logged actions rather than revenue, and the framework cannot repair faulty tools or orchestration code. Evolved knowledge did transfer: freezing it and testing on other domains gave an average hold-out gain of 8.54 points across source-target pairs.
The gate compares averages. The sibling article on a matched replay gate argues that averages let grader noise through. The paper does not discuss grader noise, so whether that concern applies to KARMA's score is not established.
Goes deeper in: Agent Engineering → Deployment & Rollout → Rollback Discipline
Related explainers
- Matched replay gate for self-evolving skill memory — the same keep-or-reject idea with a paired test instead of an average
- Regression control in continual agent optimization — why optimization gains wash out without a guard against breaking earlier tasks