微信服务号

微信订阅号

2025-5-11- 18
Home > Archive>Volume 29, Issue 2, 2018 >267-282. DOI:10.13328/j.cnki.jos.005251
PDF HTML XML Export Cite reminder
True Online Natural Actor-Critic Algorithm for the Continuous Space Problem
Author:
Affiliation:

Clc Number:

TP301

Fund Project:

National Natural Science Foundation of China (61303108, 61373094, 61472262); Jiangsu College Natural Science Research Key Program (17KJA520004); Program of the Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education (Jilin University) (93K172014K04); Suzhou Industrial Application of Basic Research Program (SYG201422); Program of the Provincial Key Laboratory for Computer Information Processing Technology (Soochow University) (KJS1524); China Scholarship Council Project (201606920013)

  • Article
    • | |
  • Metrics
    • |
  • Reference
    • |
  • Related
    • |
  • Cited by
    • | |
  • Comments
    Abstract:

    Policy gradient methods have been extensively studied as a solution to the continuous space control problem. However, due to the presence of high variance in the gradient estimation, policy gradient based methods are restricted by low sample data utilization and slow convergence. Aiming at solving this problem, utilizing the framework of actor-critic algorithm, a true online incremental natural actor-critic (TOINAC) algorithm, which takes advantage of the natural gradient that is superior to conventional gradient, and is based on true online time difference (TOTD), is proposed. In the critic part of TOINAC algorithm, the efficiency of TOTD is adopted to estimate the value function, and in the actor part of TOINAC algorithm, a novel forward view is introduced to compute and estimate natural gradient. Then, eligibility traces are utilized to turn natural gradient into online estimation, thereby improving the accuracy of natural gradient and efficiency of the method. The TOINAC algorithm is used to integrate with the kernel method and normal distribution policy to tackle the continuous space problem. The simulation tests on cart pole, Mountain Car and Acrobot, which are classical benchmark tests for continuous space problem, verify the effeteness of the algorithm.

    Reference
    Related
    Cited by
Get Citation

朱斐,朱海军,刘全,陈冬火,伏玉琛.一种解决连续空间问题的真实在线自然梯度AC算法.软件学报,2018,29(2):267-282

Copy
Share
Article Metrics
  • Abstract:
  • PDF:
  • HTML:
  • Cited by:
History
  • Received:November 04,2016
  • Revised:December 13,2016
  • Online: March 30,2017
Links:Institute of Software, CASChina Computer FederationCASNSFC
You are the first2043811Visitors
Copyright: Institute of Software, Chinese Academy of Sciences Beijing ICP No. 05046678-4
Address：4# South Fourth Street, Zhong Guan Cun, Beijing 100190,Postal Code：100190
Phone：010-62562563 Fax：010-62562533 Email：jos@iscas.ac.cn
Technical Support：Beijing Qinyun Technology Development Co., Ltd.

Beijing Public Network Security No. 11040202500063