» Download Article

Authors
Yu, Tao; Zhen, Wei-Guo

Digital Object Identifier (DOI)
10.3182/20100826-3-TR-4015.00042

Page Numbers:
220-224

Index Terms
power system stabilizers (PSS),Q-learning,Q(λ) learning,eligibility traces

Abstract
Power system stabilizers (PSS) are used to generate supplementary control signals for excitation system in order to damp the low frequency power system oscillation. To overcome the drawbacks of conventional PSS (CPSS), numerous techniques have been proposed in the literature. This paper proposes an optimal control methodology based on reinforcement learning (RL) for damping the low frequency power system oscillations. Q(λ) learning is a relative new model-free RL algorithm that extends the one-step Q-learning by combining it with TD(λ) returns for general λ in a incremental way for the delayed RL problem. The Q(λ) learning makes use of TD(λ) returns as the value function estimator through a trace mechanism, and additional traces appear to help bridge the gap between frequency and recency information for heuristic events. In this paper, rotor speed deviation is adopted as the state feedback input, and the voltage deviation, power deviation and rotor speed deviation are formulated as reward function. The case study shows that both of them are very helpful to enhance the small-disturbance dynamics of power system. The multi-step Q(λ) learning can enhance convergence rate of reinforcement learning algorithm.

References

[1] Wenxin Liu, Venayagamoorthy G.K., Wunsch D.C. II. A heuristic
    dynamic programming based power system stabilizer for a
    turbogenerator in a single machine power system. Industry Applications
    Conference, 2003 38th IAS Annual Meeting. Conference Record of the
    Volume 1, 12-16 Oct. 2003 Page(s): 270-276 vol.1.
[2] M. A. Abido, Robust design of multi-machine power system stabilizers
    using simulated annealing. IEEE Transaction on Energy Conversion, vol.
    15, no.3, pp: 297-304, 2000.
[3] A. Hariri, and O.P. Malik. A fuzzy logic based power system stabilizer
    with learning ability. IEEE Transaction on Energy Conversion, vol. 11,
    no. 4, pp. 721-727, 1996.
[4] Tao Yu, Wei-Guo Zhen. A reinforcement learning approach to power
    system stabilizer [C]. Power & Energy Society General Meeting, 2009,
    PES '09. IEEE, 1-5.
[5] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction.
    Cambridge, MA: MIT Press, 1998.
[6] Q. Y. Hu and W. Y. Yue, Markov Decision Processes with Their
    Applications. New York: Springer, 2008.
[7] P. Jing and R. J. Williams, "Incremental multi-Step Q-learning,"
    Machine Learning, vol. 22, no. 1-3, pp. 283-290, Jan. 1996.
[8] J. C. H. Watkins and P. Dayan, "Q-learning," Machine Learning, vol. 8,
    no. 3-4, pp. 279-292, May 1992.
[9] R. S. Sutton, "Learning to predict by the methods of temporal
    differences," Machine Learning, vol. 3, no. 1, pp. 9-44, Aug. 1988.
[10] J. N. Tsitsiklis, "Asynchronous stochastic approximation and Q-learning,"
      Machine Learning, vol. 16, no. 3, pp. 185-202, Sep. 1994.
[11] P. Kundur, Power System Stability and Control, McGraw-Hill, 1994,
     Example 12.6, p. 813.
[12] Klein, Rogers, Moorty and Kundur. Analytical investigation of factors
     influencing PSS performance. IEEE Trans. On EC, Vol. 7, No 3,
     September 1992, pp. 382-390.