J-MOReL: offline model-based RL
A reimplementation of the MOReL offline reinforcement learning framework with the improvements I proposed in my MSc thesis. It outperforms the original on the D4RL HalfCheetah benchmark.
Background
MOReL (Kidambi et al., NeurIPS 2020) is an offline, model-based reinforcement learning method. It learns a model of the environment from a fixed dataset, then trains a policy inside a pessimistic version of that model: transitions the model is unsure about end the episode with a penalty, so the policy cannot exploit the model’s mistakes.
J-MOReL is a clean, modular reimplementation of MOReL, extended with the improvements I proposed in my MSc thesis.
Improvements
- Better dynamics ensembles. Deeper networks, learning-rate schedulers, elite ensemble members, and networks that predict the parameters of a Gaussian instead of a single next state. The ensembles come from my dynamics-ensembles-rl library.
- Observation normalisation across every component.
- Disagreement scaling. A Jacobian-variance-based scaling that makes the ensemble’s disagreement a more reliable measure of uncertainty.
Reproducibility
Trained ensembles are published on Hugging Face, and experiments are staged and tracked with Guild AI, which also works on SLURM clusters. The README includes practical guidance for tuning the pessimism coefficient.
