Features extracted from auto-encoder based echo state network and its applications to process modeling

doi:10.11949/0438-1157.20191350

Abstract

Abstract:

In the process of complex chemical modeling, the traditional static neural network modeling can not meet certain accuracy because of the time sequence, high nonlinearity and high dimension of process data. To solve this problem, a feature extracted from auto-encoder based echo state network (FEAE-ESN) is proposed in this paper. In the traditional echo state network (ESN) method, the number of nodes in the reserve pool of ESN is large, and then the dimension of the reserve pool output is very high. Therefore, to solve this problem, the auto-encoder is used to extract features from the reserve pool output of well-trained ESN. Through the feature extraction of auto-encoder, on one hand, the dimension of the output of the reserve pool can be effectively reduced, thereby reducing the complexity of the data; on the other hand, the collinearity of the outputs of the original reserve pool can be removed through extracting features, which can further improve the calculation performance of generalized inverse. Ultimately, the modeling accuracy of ESN is improved. The proposed FEAE-ESN is applied to modeling the Tennessee-Eastman process. The simulation results verify the effectiveness of the proposed method.

Key words: auto-encoder, echo state network, feature extraction, soft sensor, process modeling

摘要：

在复杂化工建模过程中，由于过程数据的时序性、高非线性以及高维数的特点，导致传统的静态神经网络建模无法满足一定的精度。为了解决该问题，提出一种基于自编码神经网络特征提取的回声状态网络模型（features extracted from auto-encoder based echo state network，FEAE-ESN）。传统回声状态网络(echo state network, ESN)方法中，储备池的节点数目很多，输出的维数很高，数据间存在共线性。为解决上述问题，待回声状态网络训练好之后，使用自编码神经网络对其储备池输出进行特征提取。通过自编码网络特征提取，一方面可以有效地降低储备池输出的维数，从而降低数据的复杂度；另一方面提取的特征去除了原有储备池输出的共线性，能够进一步提高广义逆的计算性能；最终提高回声状态网络的建模精度。所提方法FEAE-ESN用于田纳西-伊斯曼复杂过程建模，仿真结果验证了所提方法的有效性。

关键词: 自编码神经网络, 回声状态网络, 特征提取, 软测量, 过程建模

CLC Number:

TP 29

Bao ZHU, Junfei QIAO. Features extracted from auto-encoder based echo state network and its applications to process modeling[J]. CIESC Journal, 2019, 70(12): 4770-4776.

朱宝, 乔俊飞. 基于自编码神经网络特征提取的回声状态网络研究及过程建模应用[J]. 化工学报, 2019, 70(12): 4770-4776.

Figures/Tables 10

Fig.1 Structure of ESN

Fig.2 Structure of auto-encoder

Fig.3 Structure of the proposed FEAE-ESN

Fig.4 Procedure of establishing the proposed FEAE-ESN

Table 1 Information of inputs of Tennessee-Eastman process

No.	Variable description
1	A feed
2	D feed
3	E feed
4	A and C feed
5	recycle flow
6	reactor feed rate
7	reactor temperature
8	purge rate
9	product separator temperature
10	product separator pressure
11	product separator underflow
12	stripper pressure
13	stripper temperature
14	stripper steam flow
15	reactor cooling water outlet temperature
16	separator cooling water outlet temperature

Fig.5 Comparison of expected values and predictions using ESN

Fig.8 Distributions of testing residual errors of FEAE-ESN

Fig.6 Distributions of testing residual errors of ESN

Fig.7 Comparison of expected values and predictions using FEAE-ESN

Table 2 Modeling accuracy comparison between ESN and FEAE-ESN

方法	网络测试RMSE
ESN	0.0615
FEAE-ESN	0.0207

References 30

1	朱群雄, 张晓晗, 顾祥柏, 等. 基于特征提取的函数连接神经网络研究及其化工过程建模应用[J]. 化工学报, 2018, 69(3): 907-912.
	Zhu Q X, Zhang X H, Gu X B, et al. Research and application of feature extraction derived functional link neural network[J]. CIESC Journal, 2018, 69(3): 907-912.
2	Binois M, Gramacy R B, Ludkovski M. Practical heteroscedastic Gaussian process modeling for large simulation experiments[J]. Journal of Computational and Graphical Statistics, 2018, 27(4): 808-821.
3	Li W, Zhao C, Gao F. Linearity evaluation and variable subset partition based hierarchical process modeling and monitoring[J]. IEEE Transactions on Industrial Electronics, 2017, 65(3): 2683-2692.
4	Wu F Y, Asada H H. Implicit and intuitive grasp posture control for wearable robotic fingers: a data-driven method using partial least squares [J]. IEEE Transactions on Robotics, 2016, 32(1): 1-11.
5	Li H D, Xu Q S, Liang Y Z. libPLS: an integrated library for partial least squares regression and linear discriminant analysis[J]. Chemometrics and Intelligent Laboratory Systems, 2018, 176: 34-43.
6	Awad M, Khanna R. Support vector regression [J]. Neural Information Processing Letters & Reviews, 2007, 11(10): 203-224.
7	Bourinet J M. Rare-event probability estimation with adaptive support vector regression surrogates[J]. Reliability Engineering & System Safety, 2016, 150: 210-221.
8	徐圆, 张伟, 张明卿, 等. 基于FEEMD-AE与反馈极限学习机组合模型预测研究与应用[J]. 化工学报, 2018, 69(3): 1064-1070.
	Xu Y, Zhang W, Zhang M Q, et al. Prediction research and application of a combination model based on FEEMD-AE and feedback extreme learning machine[J]. CIESC Journal, 2018, 69(3): 1064-1070.
9	Tang J, Deng C, Huang G B. Extreme learning machine for multilayer perceptron[J]. IEEE Transactions on Neural Networks and Learning Systems, 2015, 27(4): 809-821.
10	Sadeghi B H M. A BP-neural network predictor model for plastic injection molding process[J]. Journal of Materials Processing Technology, 2000, 103(3): 411-416.
11	Ren T, Liu S, Yan G, et al. Temperature prediction of the molten salt collector tube using BP neural network[J]. IET Renewable Power Generation, 2016, 10(2): 212-220.
12	Wu J, Long J, Liu M. Evolving RBF neural networks for rainfall prediction using hybrid particle swarm optimization and genetic algorithm[J]. Neurocomputing, 2015, 148: 136-142.
13	赵忠盖, 刘飞. 动态因子分析模型及其在过程监控中的应用[J]. 化工学报, 2009, 60(1): 183-186.
	Zhao Z G, Liu F. Modeling using dynamic factor analysis and its application in process monitoring[J]. CIESC Journal, 2009, 60(1): 183-186.
14	Ghiassi M, Saidane H, Zimbra D K. A dynamic artificial neural network model for forecasting time series events [J]. International Journal of Forecasting, 2005, 21(2): 341-362.
15	Du X, Vasudevan R, Johnson-Roberson M. Bio-LSTM: a biomechanically inspired recurrent neural network for 3-D pedestrian pose and gait prediction [J]. IEEE Robotics and Automation Letters, 2018, 4(2): 1501-1508.
16	Zen H, Sak H. Unidirectional long short-term memory recurrent neural network with recurrent output layer for low-latency speech synthesis[C]//2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2015: 4470-4474.
17	Feng W, Wu Y, Fan Y. A new method for the prediction of network security situations based on recurrent neural network with gated recurrent unit[J]. International Journal of Intelligent Computing and Cybernetics, 2018, 11(6): 511-525.
18	Jaeger H. The “echo state” approach to analysing and training recurrent neural networks—with an erratum note[R].Bonn: German National Research Center for Information Technology GMD Technical Report, 2001.
19	Aboelmaged M G. Knowledge sharing through enterprise social network (ESN) systems: motivational drivers and their impact on employees productivity[J]. Journal of Knowledge Management, 2018, 22(2): 362-383.
20	Gillum T L, George R H, Leitmeyer J E. An autoencoder for clinical and regulatory data processing [J]. Drug Information Journal, 1995, 29(1): 107-113.
21	Kodirov E, Xiang T, Gong S. Semantic autoencoder for zero-shot learning[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 3174-3183.
22	Zhang Z, Song Y, Qi H. Age progression/regression by conditional adversarial autoencoder[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 5810-5818.
23	彭荻, 贺彦林, 徐圆, 等. 基于数据特征提取的AANN-ELM研究及化工应用[J]. 化工学报, 2012, 63(9): 2920-2925.
	Peng D, He Y L, Xu Y, et al. Research and chemical application of data feature extraction based AANN-ELM neural network[J]. CIESC Journal, 2012, 63(9): 2920-2925.
24	才轶, 徐圆, 朱群雄, 等. 基于自联想神经网络的数据滤波功能与应用[J]. 计算机与应用化学, 2009, 26(5): 673-676.
	Cai Y, Xu Y, Zhu Q X, et al. Data filtering method and application based on auto-associative neural network[J]. Computers and Applied Chemistry, 2009, 26(5): 673-676.
25	Hou X, Shen L, Sun K, et al. Deep feature consistent variational Autoencoder [C]//2017 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2017: 1133-1141.
26	Yu J, Hong C, Rui Y, et al. Multitask autoencoder model for recovering human poses [J]. IEEE Transactions on Industrial Electronics, 2017, 65(6): 5060-5068.
27	Fu H, Lei P, Tao H, et al. Improved semi-supervised autoencoder for deception detection [J]. PloS ONE, 2019, 14(10): e0223361.
28	Co-Reyes J D, Liu Y X, Gupta A, et al. Self-consistent trajectory autoencoder: hierarchical reinforcement learning with trajectory embeddings [C]//Proceedings of the 35th International Conference on Machine Learning. Stockholm, Sweden, 2018.
29	Zhu Q X, Meng Q Q, He Y L. Novel multidimensional feature pattern classification method and its application to fault diagnosis [J]. Industrial & Engineering Chemistry Research, 2017, 56(31): 8906-8916.
30	Xu Y, Shen S Q, He Y L, et al. A novel hybrid method integrating ICA-PCA with relevant vector machine for multivariate process monitoring [J]. IEEE Transactions on Control Systems Technology, 2018, 27(4): 1780-1787.

[1]	Linqi YAN, Zhenlei WANG. Multi-step predictive soft sensor modeling based on STA-BiLSTM-LightGBM combined model [J]. CIESC Journal, 2023, 74(8): 3407-3418.
[2]	Weiming SHAO, Wenxue HAN, Wei SONG, Yong YANG, Can CHEN, Dongya ZHAO. Dynamic soft sensor modeling method based on distributed Bayesian hidden Markov regression [J]. CIESC Journal, 2023, 74(6): 2495-2502.
[3]	Le ZHOU, Chengkai SHEN, Chao WU, Beiping HOU, Zhihuan SONG. Deep fusion feature extraction network and its application in chemical process soft sensing [J]. CIESC Journal, 2022, 73(7): 3156-3165.
[4]	LIU Cong, XIE Li, YANG Huizhong. Multi-model soft sensor development for penicillin fermentation process based on improved density peak clustering [J]. CIESC Journal, 2021, 72(3): 1606-1615.
[5]	Yuhao DU, Gaowei YAN, Rong LI, Fang WANG. Multiple working conditions soft sensor modeling method of geodesic flow kernel based on locally linear embedding [J]. CIESC Journal, 2020, 71(3): 1278-1287.
[6]	Yanlin HE, Ye TIAN, Xiangbai GU, Yuan XU, Qunxiong ZHU. Regularization based functional link neural network and its applications to modeling complex chemical processes [J]. CIESC Journal, 2020, 71(3): 1072-1079.
[7]	YANG Yijun,WANG Zhenlei,WANG Xin. Soft sensor modeling method based on hybrid model of nearest neighbor and neural network [J]. CIESC Journal, 2020, 71(12): 5696-5705.
[8]	WANG Yaxin,XU Baochang,XU Chaonong,DONG Xiujuan,XU Liwei. Attention LSTM network identification method based on factory data [J]. CIESC Journal, 2020, 71(12): 5664-5671.
[9]	Xuezhi DAI,Weili XIONG. A fast active learning method based on kernel extreme learning machine and its application for soft sensing [J]. CIESC Journal, 2020, 71(11): 5226-5236.
[10]	Xiaoqin LIAN, Liwei WANG, Sa AN, Wei WEI, Zaiwen LIU. On soft sensor of chemical oxygen demand by SOM-RBF neural network [J]. CIESC Journal, 2019, 70(9): 3465-3472.
[11]	Jing WU, Yiqi LIU, Jian LIU, Daoping HUANG, Yu QIU, Guangping YU. Study on the soft sensor of multi-kernel relevance vector machine based on time difference [J]. CIESC Journal, 2019, 70(4): 1472-1484.
[12]	Wenpeng JI, Huizhong YANG. Multi-manifold soft sensor based on modified expanding search clustering algorithm [J]. CIESC Journal, 2019, 70(2): 723-729.
[13]	Dejian LI, Haoran LIU, Bin LIU, Zeren LIU, Weitao WANG, Yan WEN. Predictive control of free calcium oxide based on improved echo state network [J]. CIESC Journal, 2019, 70(12): 4749-4759.
[14]	Rongrong ZHAO, Zhonggai ZHAO, Fei LIU. Gaussian process regression modeling of fermentation process based on k-nearest neighbor mutual information [J]. CIESC Journal, 2019, 70(12): 4741-4748.
[15]	QIU Yu, LIU Yiqi, WU Jing, HUANG Daoping. A self-adaptive multi-output soft sensor modeling based on deep neural network [J]. CIESC Journal, 2018, 69(7): 3101-3113.