A. K. Mackay
Contents

A.2.3

Appendix A.2 Reinforcement learning

MBiCEM: planning on learned models

A trajectory planner that runs iCEM on learned dynamics models and heavily penalises the predictions the model is unsure about.

2025

Part of my MSc thesis

PyTorch, MuJoCo, D4RL

Approach

iCEM (Pinneri et al., CoRL 2020) is a sampling-based planner: at every step it simulates many candidate action sequences, keeps the best one and applies only its first action. MBiCEM runs it on a learned dynamics ensemble instead of a perfect simulator. A learned model can be confidently wrong, so simulated trajectories on which the ensemble members disagree strongly are heavily penalised.

  • Reimplements iCEM and adapts it to learned models.
  • Supports MBOP (Argenson et al., NeurIPS 2020) as an alternative planner.
  • Explores ensembles of value functions and behaviour-cloned policies to guide planning.

Result

On halfcheetah-medium-expert, the uncertainty penalty is the difference between a planner that stalls and one that keeps improving throughout the episode.

Planning loop diagram: observe the state, simulate candidate action sequences with a neural network model, select the best trajectory, apply its first action to the environment, and repeat.
The planning loop: observe the state, simulate candidate action sequences with the learned model, select the best trajectory and apply only its first action.
Line chart of return over 1,000 timesteps. With the disagreement penalty the return rises to about 6,700; without it the return stalls near 1,800.
Return over one episode on halfcheetah-medium-expert. Without the disagreement penalty the planner stalls near 1,800; with it, the return keeps rising to about 6,700.