Structure design for recurrent RBF neural network based on recursive orthogonal least squares

doi:10.11949/j.issn.0438-1157.20170771

Abstract

Abstract:

Aiming at the problem of recurrent radial basis function (RRBF) neural network structure which is difficult to be self-adaptive, this paper proposes a structure design method based on recursive orthogonal least square (ROLS) algorithm. Firstly, ROLS algorithm is used to calculate the contribution and the loss function of hidden layer neurons, which determines to increase or be grouped into inactive neurons, and the topology structure of neural network is adjusted accordingly. At the same time, singular value decomposition (SVD) is applied to determine the best number of hidden layer neurons in order to delete the neurons of the inactive group, which effectively solves the problems of RRBF neural network structure which is redundant and hardly self-adaptive. Secondly, the gradient descent algorithm is utilized to update the parameters of RRBF neural network in order to ensure the accuracy of neural network. Finally, several experiments including the Mackey-Glass time series prediction, nonlinear system identification and key water quality parameters dynamic modeling in wastewater treatment process are conducted, and the simulation results prove the feasibility and effectiveness of the structure design method.

Key words: neural network, structure design, algorithm, singular value decomposition, dynamic modeling

摘要：

针对递归RBF神经网络结构难以自适应问题，提出一种基于递归正交最小二乘（recursive orthogonal least squares，ROLS）算法的结构设计方法。首先，利用ROLS算法来计算隐含层神经元的独立贡献度和损失函数，以此判断增加或归为不活跃组的神经元，同时调整神经网络的拓扑结构，并且利用奇异值分解（singular value decomposition，SVD）决定最佳的隐含层神经元个数，以此来删除不活跃组中相对不活跃的神经元，有效地解决了递归RBF神经网络结构冗余和难以自适应问题。其次，利用梯度下降算法更新递归RBF神经网络的参数来保证神经网络的精度。最后，通过对Mackey-Glass时间序列预测、非线性系统辨识和污水处理过程中关键水质参数动态建模，证明了该结构设计方法的可行性和有效性。

关键词: 神经网络, 结构设计, 算法, 奇异值分解, 动态建模

CLC Number:

TP183

QIAO Junfei, MA Shijie, YANG Cuili. Structure design for recurrent RBF neural network based on recursive orthogonal least squares[J]. CIESC Journal, 2018, 69(3): 1191-1199.

乔俊飞, 马士杰, 杨翠丽. 基于ROLS算法的递归RBF神经网络结构设计[J]. 化工学报, 2018, 69(3): 1191-1199.

References 31

[1]	YE J, GE L D, WU Y X. An application of improved RBF neural network in modulation recognition[J]. Acta Automatica Sinica, 2007, 33(6):652-654.
[2]	CHEN S, WANG X X, BROWN D J. Sparse incremental regression modeling using correlation criterion with boosting search[J]. IEEE Signal Processing Letters, 2005, 12(3):198-201.
[3]	CHEN S, WOLFGANG A, HARRIS C J, et al. Symmetric RBF classifier for nonlinear detection in multiple-antenna-aided systems[J]. IEEE Transactions on Neural Networks, 2008, 19(5):737-745.
[4]	CARA A, POMARES H, ROJAS I. A new methodology for the online adaptation of fuzzy self-structuring controllers[J]. IEEE Transactions on Fuzzy Systems, 2011, 19(3):449-464.
[5]	FAN H J, SONG Q. A linear recurrent kernel online learning algorithm with sparse updates[J]. Neural Networks, 2014, 50(3):142-153.
[6]	NIU B, LI L. Adaptive neural network tracking control for a class of switched strict feedback nonlinear systems with input delay[J]. Neurocomputing, 2016, 173(3):2121-2128.
[7]	HAN H G, LI Y, GUO Y N, et al. A soft computing method to predict sludge volume index based on a recurrent self-organizing neural network[J]. Applied Soft Computing, 2016, 38(1):477-486.
[8]	ZHANG H, WANG Z, LIU D. A comprehensive review of stability analysis of continuous-time recurrent neural networks[J]. IEEE Transactions on Neural Networks and Learning Systems, 2014, 25(7):1229-1262.
[9]	LEE W, LEE S, PARK P. Improved stability criteria for recurrent neural networks with interval time-varying delays via new Lyapunov functionals[J]. Neurocomputing, 2015, 155(1):128-134.
[10]	XIE T, YU H, HEWLETT J, et al. Fast and efficient second-order method for training radial basis function networks[J]. IEEE Transactions on Neural Networks and Learning Systems, 2012, 23(4):609-619.
[11]	WANG X, YU J, LI C, et al. Robust stability of stochastic fuzzy delayed neural networks with impulsive time window[J]. Neural Networks, 2015, 67(1):84-91.
[12]	HAN H G, WU X L, QIAO J F. Nonlinear systems modeling based on self-organizing fuzzy-neural-network with adaptive computation algorithm[J]. IEEE Transactions on Cybernetics, 2014, 44(4):554-564.
[13]	MENG D, PEI Z. Dynamic adaptive learning algorithm based on two-fuzzy neural-networks[J]. Neurocomputing, 2014, 125(1):88-94.
[14]	CRUZ D, MAIA R, SILVA L, et al. A bee-inspired data clustering approach to design RBF neural network classifiers[J]. Neurocomputing, 2016, 172(3):427-437.
[15]	KARAYIANNNIS N B, MI G W. Growing radial basis neural networks:merging supervised and unsupervised learning with network growth techniques[J]. IEEE Transactions on Neural Network, 1997, 8(6):1492-1506.
[16]	SUBRAHMANYA N, SHIN Y C. Constructive training of recurrent neural networks using hybrid optimization[J]. Neurocomputing, 2010, 73(13):2624-2631.
[17]	HSU C F. Adaptive backstepping Elman-based neural control for unknown nonlinear systems[J]. Neurocomputing, 2014, 136(1):170-179.
[18]	ZHENG P S, DIMITRAKAKIS C, TRIESCH J. Network self-organization explains the statistics and dynamics of synaptic connection strengths in cortex[J]. Computational Biology, 2013, 9(1):1-8.
[19]	LEUNG C S, LAM P M. A local training-pruning approach for recurrent neural networks[J]. International Journal of Neural Systems, 2003, 13(1):25-38.
[20]	乔俊飞, 韩红桂. RBF神经网络的结构动态优化设计[J]. 自动化学报, 2010, 36(6):865-872. QIAO J F, HAN H G. Optimal structure design for RBFNN structure[J]. Acta Automatica Sinica, 2010, 36(6):865-872.
[21]	TOK D K, YU D L, MATHEWS C. Adaptive structure radial basis function network model for processes with operating region migration[J]. Neurocomputing, 2015, 155:186-193.
[22]	GOMM J B, YU D L. Selecting radial basis function network centers with recursive orthogonal least squares training[J]. IEEE Transactions on Neural networks, 2000, 11(2):306-314.
[23]	BROOMHEAD D S, LOWE D. Multivariable functional interpolation and adaptive networks[J]. Complex Systems, 1988, 2:321-355.
[24]	KAMINSKI W, STRUMILLO P. Kernel orthonomalization in radial basis function neural networks[J]. IEEE Transactions on Neural Networks, 1997, 8(5):1177-1183.
[25]	AKAIKE H. Fitting autoregressive models for prediction[J]. Annals of the Institute of Statistical Mathematics, 1969, 21:425-439.
[26]	TEOH E J, TAN K C, XIANG C. Estimating the number of hidden neurons in a feedforward network using the singular value decomposition[J]. IEEE Transactions on Neural Networks, 2006, 17(6):1623-1629.
[27]	KONSTANTINIDES K, YAO K. Statistical analysis of effective singular values in matrix rank determination[J]. IEEE Transactions on Acoustics, Speech Signal Process, 1988, 36(5):757-763.
[28]	HAN H G, CHEN Q L, QIAO J F. An efficient self-organizing RBF neural network for water quality prediction[J]. Neural Networks, 2011, 24(7):717-725.
[29]	韩红桂, 乔俊飞, 薄迎春. 基于信息强度的RBF神经网络结构设计研究[J]. 自动化学报, 2012, 38(7):1083-1090. HAN H G, QIAO J F, BO Y C. On structure design for RBF neural network based on information strength[J]. Acta Automatica Sinica, 2012, 38(7):1083-1090.
[30]	乔俊飞, 安茹, 韩红桂. 基于RBF神经网络的出水氨氮预测研究[C]//杨辉. 第26届中国过程控制会议. 南昌:中国自动化学会, 2015:120-127. QIAO J F, AN R, HAN H G. Based on the RBF neural network water ammonia nitrogen prediction research[C]//YANG H. The 26th Process Control Meeting in China. Nanchang:China Society of Automation, 2015:120-127.
[31]	杨琴, 谢淑云. BP神经网络在洞庭湖氨氮浓度预测中的应用[J]. 水资源与水工程学报, 2006, 17(1):65-67. YANG Q, XIE S Y. Application of BP neural network into predicting NH3-N concentration of Dongting Lake[J]. Journal of Water Resources & Water Engineering, 2006, 17(1):65-70.