A multi-step Q(λ) learning approach to power system stabilizer
Adaptation and Learning in Control and Signal Processing, Volume # 10 | Part# 1
Yu, Tao; Zhen, Wei-Guo
Digital Object Identifier (DOI)
power system stabilizers (PSS),Q-learning,Q(λ) learning,eligibility traces
Power system stabilizers (PSS) are used to generate supplementary control signals for excitation system in order to damp the low frequency power system oscillation. To overcome the drawbacks of conventional PSS (CPSS), numerous techniques have been proposed in the literature. This paper proposes an optimal control methodology based on reinforcement learning (RL) for damping the low frequency power system oscillations. Q(λ) learning is a relative new model-free RL algorithm that extends the one-step Q-learning by combining it with TD(λ) returns for general λ in a incremental way for the delayed RL problem. The Q(λ) learning makes use of TD(λ) returns as the value function estimator through a trace mechanism, and additional traces appear to help bridge the gap between frequency and recency information for heuristic events. In this paper, rotor speed deviation is adopted as the state feedback input, and the voltage deviation, power deviation and rotor speed deviation are formulated as reward function. The case study shows that both of them are very helpful to enhance the small-disturbance dynamics of power system. The multi-step Q(λ) learning can enhance convergence rate of reinforcement learning algorithm.
 Wenxin Liu, Venayagamoorthy G.K., Wunsch D.C. II. A heuristic dynamic programming based power system stabilizer for a turbogenerator in a single machine power system. Industry Applications Conference, 2003 38th IAS Annual Meeting. Conference Record of the Volume 1, 12-16 Oct. 2003 Page(s): 270-276 vol.1.  M. A. Abido, Robust design of multi-machine power system stabilizers using simulated annealing. IEEE Transaction on Energy Conversion, vol. 15, no.3, pp: 297-304, 2000.  A. Hariri, and O.P. Malik. A fuzzy logic based power system stabilizer with learning ability. IEEE Transaction on Energy Conversion, vol. 11, no. 4, pp. 721-727, 1996.  Tao Yu, Wei-Guo Zhen. A reinforcement learning approach to power system stabilizer [C]. Power & Energy Society General Meeting, 2009, PES '09. IEEE, 1-5.  R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press, 1998.  Q. Y. Hu and W. Y. Yue, Markov Decision Processes with Their Applications. New York: Springer, 2008.  P. Jing and R. J. Williams, "Incremental multi-Step Q-learning," Machine Learning, vol. 22, no. 1-3, pp. 283-290, Jan. 1996.  J. C. H. Watkins and P. Dayan, "Q-learning," Machine Learning, vol. 8, no. 3-4, pp. 279-292, May 1992.  R. S. Sutton, "Learning to predict by the methods of temporal differences," Machine Learning, vol. 3, no. 1, pp. 9-44, Aug. 1988.  J. N. Tsitsiklis, "Asynchronous stochastic approximation and Q-learning," Machine Learning, vol. 16, no. 3, pp. 185-202, Sep. 1994.  P. Kundur, Power System Stability and Control, McGraw-Hill, 1994, Example 12.6, p. 813.  Klein, Rogers, Moorty and Kundur. Analytical investigation of factors influencing PSS performance. IEEE Trans. On EC, Vol. 7, No 3, September 1992, pp. 382-390.