Soft Actor-Critic for railway optimal maintenance planning under partial observability

Abstract

The optimal maintenance planning for railway systems forms a complex sequential decision-making problem. Optimal maintenance actions ought to be configured on the basis of updated rail condition estimates. To this end, structural health monitoring solutions can be used for reliably tracking the condition of railway infrastructure. However, the measurements gathered from continuous monitoring can only offer incomplete, often noise-corrupted, information of the real condition states, which implies the need for decision-making under uncertainty. For tackling the inherent uncertainty, the problem can be formalized as a Partially Observable Markov Decision Process (POMDP). Two families of methods are generally used to solve such formulations, namely Dynamic Programming (DP) and Reinforcement Learning (RL). In this work, we apply deep RL to solve a real-world railway maintenance planning problem modeled as a POMDP without assuming any knowledge of the problem parameters, in order to derive a full model-free solution. In particular, we employ the Soft Actor-Critic method, extended to partial observability, and compare the quality of the solution against classical DP methods analyzed in previous works.