Foundry Labs / From the community
Offline Reinforcement Learning: Ask What Your Historical Data Cannot Show
In this article
Offline reinforcement learning promises to learn a policy from existing records without experimenting in the live environment. That is attractive when exploration is costly or unsafe. At Tokyo AI's offline RL session, Johannes Ackermann and Shivakanth Sujit focused attention on what those records can and cannot prove.
Ask how the logs were produced
Johannes's talk examined datasets with structured non-stationarity. A log is not a neutral sample of all possible actions and outcomes. It was produced by a policy, under conditions that may have changed over time. If operators, equipment or incentives shifted, pooling every record as though it came from one stable process can mislead a learner. Before training, a team should map the periods, policies and context that produced its data.
Offline learning also cannot directly observe what would have happened after an action the historical policy rarely or never took. A model may appear confident in precisely the region where the data offer little support. That is a reason to bound the candidate policy and evaluate it against the coverage of the logs, not to infer that a good training score implies safe live behavior.
Evaluate the evaluation method
Shivakanth's talk examined sample efficiency and the way offline RL algorithms are compared. Training for the same number of gradient steps does not necessarily compare methods fairly when data volume changes. The evaluation protocol should state what is held constant and what cost is being measured. Otherwise the team may select a method based on an artifact of the test setup.
Consider an illustrative warehouse routing policy. Historical logs might cover normal demand but very few closures or equipment faults. A research pilot can use those logs to compare candidate policies, identify weak coverage and design a cautious validation step. It should not send a new policy into live operations merely because it performs well on replayed data. The useful outcome of the pilot may be learning which situations still need better data or human control.
Tokyo AI is a community of builders, researchers and business leaders in Japan, run by Foundry Labs. See the event page.
Materials from the talks by Johannes Ackermann and Shivakanth Sujit are available upon request. Please use our contact form and include the event title and speaker name.
Speakers
- Johannes Ackermann, researcher in offline reinforcement learning at the University of Tokyo.
- Shivakanth Sujit, researcher in reinforcement-learning evaluation.