Markov decision processes and their applications on multi-unit maintenance systems

Abstract

In this paper an extensive mathematical research into Markov decision processes was conducted. First, we presented the necessary preliminary knowledge on regular Markov chains and stochastic processes. This knowledge was then extended into Markov decision processes by incorporating decision moments, possible actions, time-horizons, rewards and more. We focused on the different types of time-horizons as well as rewards and how they can be derived. Key results concerning the existence and uniqueness of optimal policies and the derivation of maximal rewards were established through proof such as the Banach fixed point theorem. In doing so, we provided both the theoretical justification and the computational steps that would prove to be necessary for practical applications. Continuing with this concept we discussed Markov renewal theory; a more practical form of Markov processes which incorporates the number of times a process has been in a certain state of interest. The final part of the paper applied these tools to multi-unit maintenance systems. We studied two research papers by Salari and Makis as well as Mercier and Castro applying the aforementioned theory to maintenance systems. In the first case, semi-Markov decision processes were used to derive a long-run average cost function for a maintenance system with N identical units, subject to deterioration. This allowed for an iteration algorithm to find the optimal policy for a multi-unit maintenance system, such as wind farms or solar panel fields. The research by Mercier and Castro followed by observing a continuously monitored production system, where the rate of deterioration is γ-distributed. Similarly, their goal was to act proactively, and initiate maintenance at an optimal moment to reduce costs. They eventually managed to show that one can indeed relax the ‘as good as new’ assumption within a maintenance system using Markov renewal theory and Monte Carlo integration. Combining these two research papers allowed us to suggest a potential theoretical framework which could be used in future studies.