Concepts

Tracks, environments & metrics

The benchmark vocabulary: tracks, environments, agents, and the metrics that rank them — including confidence intervals, model fidelity, and planning cost.

Tracks

A track selects which seed suite a run is evaluated against. Seed suites are fixed in code so results are reproducible.

Track	What it measures
`train`	Procedural train seeds — the in-distribution baseline.
`test`	Held-out procedural test seeds — the headline generalization number.
`continual`	Nonstationary dynamics that shift difficulty every few episodes.

Generalization gap

The gap is train_success_rate − test_success_rate. A small gap means the agent generalizes; a large gap means it overfit the train seeds.

Environments

Environment	Tests
`memory_maze`	Partial observability + memory: navigate a maze where the goal is out of view.
`switch_quest`	Causal sub-goals: toggle switches in the right order to unlock a reward.
`craft_lite`	Compositional achievements: gather and combine resources to complete a craft tree.

Agents

Baselines span scripted oracles to learned and planning agents. The registry ships: random, greedy_oracle, planner_oracle, imagination_mpc, search_mcts, and ppo.

Custom agents

An agent implements reset(seed), act(obs, info), and observe(transition), with an optional get_trace() for planner debug output. Register it in agents/worldmodel_agents/registry.py.

Metrics

Runs report scalar metrics plus structured sub-objects. The leaderboard ranks on success rate but surfaces uncertainty and cost alongside it.

Metric	Meaning
`success_rate`	Fraction of episodes solved — the primary ranking metric.
`mean_return`	Average episode return.
`success_rate_ci`	A `[low, high]` confidence interval, rendered as a whisker on the leaderboard.
`median_steps_to_success`	Sample efficiency — how quickly the agent reaches the goal.
`model_fidelity`	k-step reward error of the world model at `k1 / k5 / k20` horizons.
`planning_cost`	Compute cost, including `wall_clock_ms_per_step` and imagined transitions.
`generalization_gap`	Train vs. test success-rate delta.

Why confidence intervals matter

Two agents with the same point success rate are not equal if one has a far wider interval. The leaderboard always renders the CI so rankings stay honest about sample size and variance.

The continual track adds transfer metrics: forward_transfer, backward_transfer, and forgetting.