Anomaly detection of process unit based on LSTM time series reconstruction

doi:10.11949/j.issn.0438-1157.20181050

Abstract

Abstract:

Industrial production equipment usually sets sensor alarm thresholds for alarms, but it is difficult to capture time series abnormalities below the alarm thresholds. The traditional statistics based detection method has great challenges in these time series anomaly detection. In this paper, an approach to the anomaly detection of process units is proposed by using the long short term memory (LSTM) time series reconstruction. At first, an LSTM network is introduced to vectorize the time series of sensor data, and another LSTM network is utilized to reconstruct the time series in reverse sequence. Then, the errors between the reconstructed values and the actual values are used to estimate the anomaly probability by the maximum likelihood estimation method. Eventually, anormaly detection is achieved by learning the abnormal alarm thresholds. Simulation resutls on the ECG standard testing data, energy data and sensor data of the dangerous goods tank have shown the effectiveness of the proposed method on data with different lengths.

Key words: algorithm, neural networks, parameter estimation, LSTM, time series, anomaly detection, maximum likelihood estimation

摘要：

工业生产装置通常设置传感器报警阈值进行报警，但是对处于报警阈值以下的时间序列异常难以及时捕捉。基于统计的传统检测方法在解决时间序列异常检测上存在很大挑战，因此提出基于long short term memory (LSTM)时间序列重建的方法进行生产装置的异常检测。该算法首先引入一层LSTM网络对传感器数据的时间序列进行向量表示，采用另一层LSTM网络对时间序列进行逆序重建，然后利用重建值与实际值之间的误差，通过极大似然估计方法对该段序列进行异常概率估计，最终通过学习异常报警阈值实现时间序列异常检测。采用ECG测试数据、能源数据与危险品储罐传感器数据进行了仿真实验，验证了所提方法在不同长度的数据上的有效性。

关键词: 算法, 神经网络, 参数估计, LSTM, 时间序列, 异常检测, 极大似然估计

CLC Number:

TP 391

Shan DOU, Guangyu ZHANG, Zhihua XIONG. Anomaly detection of process unit based on LSTM time series reconstruction[J]. CIESC Journal, 2019, 70(2): 481-486.

窦珊, 张广宇, 熊智华. 基于LSTM时间序列重建的生产装置异常检测[J]. 化工学报, 2019, 70(2): 481-486.

Figures/Tables 7

Fig.1 LSTM memory block

Fig.2 Overall structure

Table 1 Procedure of MLE parameter estimation[25]

1.设异常概率 $p$ 随重建误差e呈高斯分布，则其似然函数如式(4)所示；

2.对似然函数取对数并求和，整理后为

$H (μ, Σ) = ∑ e i ∈ V 1 l n p (e i | μ, Σ)$ (8)

3.对 $H μ, Σ$ 求偏导，使得其满足：

$? H (μ, Σ) ? μ = 0 ? H (μ, Σ) ? Σ = 0$

4.求解上述方程得到参数估计值如下所示，其中N为数据集 $N V 1$ 的时间点总数。

$μ = 1 N ∑ i = 1 N e i Σ = 1 N ∑ i = 1 N (e i - μ) (e i - μ) T$ (9)

Table 1 Procedure of MLE parameter estimation[25]

1.设异常概率 $p$ 随重建误差e呈高斯分布，则其似然函数如式(4)所示；

2.对似然函数取对数并求和，整理后为

$H (μ, Σ) = ∑ e i ∈ V 1 l n p (e i | μ, Σ)$ (8)

3.对 $H μ, Σ$ 求偏导，使得其满足：

$? H (μ, Σ) ? μ = 0 ? H (μ, Σ) ? Σ = 0$

4.求解上述方程得到参数估计值如下所示，其中N为数据集 $N V 1$ 的时间点总数。

$μ = 1 N ∑ i = 1 N e i Σ = 1 N ∑ i = 1 N (e i - μ) (e i - μ) T$ (9)

Fig.3 ECG standard test data

Fig.4 Energy consumption data

Fig.5 Dangerous goods storage sensor data

Table 2 Experimental results

Data	P		R		$F 0.1$
Data	RNN	LSTM	RNN	LSTM	RNN	LSTM
ECG	0.98	1.00	0.45	0.43	0.96	0.97
EN	0.89	0.94	0.10	0.12	0.83	0.88
SE	0.89	0.92	0.20	0.26	0.86	0.89

Table 2 Experimental results

Data	P		R		$F 0.1$
Data	RNN	LSTM	RNN	LSTM	RNN	LSTM
ECG	0.98	1.00	0.45	0.43	0.96	0.97
EN	0.89	0.94	0.10	0.12	0.83	0.88
SE	0.89	0.92	0.20	0.26	0.86	0.89

References 30

1	周大镯, 刘月芬, 马文秀. 时间序列异常检测[J]. 计算机工程与应用, 2008, 44(35): 145-147.
	ZhouD Z, LiuY F, MaW X. Effective time series outlier detection algorithm based on segmentation[J]. Computer Engineering and Applications, 2008, 44(35): 145-147.
2	HawkinsD M. Identification of Outliers[M]. London: Chapman and Hall, 1980: 25.
3	BillorN, HadiA S, VellemanP F. BACON: blocked adaptive computationally efficient outlier nominators[J]. Computational Statistics & Data Analysis, 2000, 34(3): 279-298.
4	KnorrE M, NgR T. A unified notion of outliers: properties and computation[C]//KDD. 1997: 219-222.
5	李钢, 周东华. 基于SPM的多变量连续过程在线故障预测方法[J]. 化工学报, 2008, 59(7): 1829-1833.
	LiG, ZhouD H. SPM-based online fault prediction approach for multivariate continuous processes[J]. Journal of Chemical Industry and Engineering(China), 2008, 59(7): 1829-1833.
6	张浩, 刘振娟, 李宏光, 等. 基于关联变量时滞分析卷积神经网络的生产过程时间序列预测方法[J]. 化工学报, 2017, 68(9): 3501-3510.
	ZhangH, LiuZ J, LiH G, et al. Process time series prediction based on application of correlated process variables to CNN time delayed analyses[J]. CIESC Journal, 2017, 68(9): 3501-3510.
7	PageE S. Continuous inspection schemes[J]. Biometrika, 1954, 41(1/2): 100-115.
8	BassevilleM, NikiforovI V. Detection of Abrupt Changes: Theory and Application[M]. Englewood Cliffs: Prentice Hall, 1993: 30.
9	MaJ, PerkinsS. Time-series novelty detection using one-class support vector machines[C]//Neural Networks, 2003. Proceedings of the International Joint Conference on. IEEE, 2003: 1741-1745.
10	王少军. 时间序列预测的可重构计算研究[D]. 哈尔滨:哈尔滨工业大学, 2012: 35.
	WangS J. Research on reconfigurable computing for time series forecasting[D]. Harbin: Harbin Institute of Technology, 2012: 35.
11	HochreiterS, SchmidhuberJ. Long short-term memory[J]. Neural Computation, 1997, 9(8): 1735-1780.
12	SutskeverI, VinyalsO, LeQ V. Sequence to sequence learning with neural networks[C]//Advances in Neural Information Processing Systems, 2014: 3104-3112.
13	MiwaM, BansalM. End-to-end relation extraction using LSTMs on sequences and tree structures[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2016: 1105-1116.
14	ChoiY, BreckE, CardieC. Joint extraction of entities and relations for opinion recognition[C]//Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2006: 431-439.
15	GravesA, SchmidhuberJ. Offline handwriting recognition with multidimensional recurrent neural networks[C]//Advances in Neural Information Processing Systems, 2009: 545-552.
16	MessinaR, LouradourJ. Segmentation-free handwritten Chinese text recognition with LSTM-RNN[C]//Document Analysis and Recognition (ICDAR), 2015 13th International Conference on. IEEE, 2015: 171-175.
17	HuangZ, XuW, YuK. Bidirectional LSTM-CRF models for sequence tagging[J]. arXivpreprint arXiv: 1508.01991, 2015.
18	MaX, HovyE. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2016: 1064-1074.
19	ChoK, van MerrienboerB, BahdanauD, et al. On the properties of neural machine translation: encoder–decoder approaches[C]//Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, 2014: 103-111.
20	ChoK, VanM B, GulcehreC, et al. Learning phrase representations using RNN encoder–decoder for statistical machine translation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014: 1724-1734.
21	CybenkoG. Approximation by superpositions of a sigmoidal function[J]. Mathematics of Control, Signals and Systems, 1989, 2(4): 303-314.
22	MaasA L, HannunA Y, NgA Y. Rectifier nonlinearities improve neural network acoustic models[J]. Proc. Icml., 2013, 30(1): 3.
23	GravesA. Long short-term memory[M]//Supervised Sequence Labelling with Recurrent Neural Networks. Heidelberg, Berlin: Springer, 2012: 37-45.
24	ChungJ, GulcehreC, ChoK H, et al. Empirical evaluation of gated recurrent neural networks on sequence modeling[J]. arXivpreprint arXiv: 1412.3555, 2014.
25	BockR D, AitkinM. Marginal maximum likelihood estimation of item parameters: application of an EM algorithm[J]. Psychometrika, 1981, 46(4): 443-459.
26	MalhotraP, VigL, ShroffG, et al. Long short term memory networks for anomaly detection in time series[C]//23rd European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN 2015. Bruges, Belgium, 2015: 89.
27	GoutteC, GaussierE. A probabilistic interpretation of precision, recall and F-score, with implication for evaluation[C]//European Conference on Information Retrieval. Heidelberg, Berlin: Springer, 2005: 345-359.
28	LeQ V, NgiamJ, CoatesA, et al. On optimization methods for deep learning[C]//Proceedings of the 28th International Conference on International Conference on Machine Learning. Omnipress, 2011: 265-272.
29	KingmaD P, BaJ. Adam: a method for stochastic optimization[J]. arXivpreprint arXiv: 1412.6980, 2014.
30	ConnorJ T, MartinR D, AtlasL E. Recurrent neural networks and robust time series prediction[J]. IEEE Transactions on Neural Networks, 1994, 5(2): 240-254.

[1]	Yue CAO, Chong YU, Zhi LI, Minglei YANG. Industrial data driven transition state detection with multi-mode switching of a hydrocracking unit [J]. CIESC Journal, 2023, 74(9): 3841-3854.
[2]	Gang YIN, Yihui LI, Fei HE, Wenqi CAO, Min WANG, Feiya YAN, Yu XIANG, Jian LU, Bin LUO, Runting LU. Early warning method of aluminum reduction cell leakage accident based on KPCA and SVM [J]. CIESC Journal, 2023, 74(8): 3419-3428.
[3]	Chengying ZHU, Zhenlei WANG. Operation optimization of ethylene cracking furnace based on improved deep reinforcement learning algorithm [J]. CIESC Journal, 2023, 74(8): 3429-3437.
[4]	Linqi YAN, Zhenlei WANG. Multi-step predictive soft sensor modeling based on STA-BiLSTM-LightGBM combined model [J]. CIESC Journal, 2023, 74(8): 3407-3418.
[5]	Xuejin GAO, Yuzhuo YAO, Huayun HAN, Yongsheng QI. Fault monitoring of fermentation process based on attention dynamic convolutional autoencoder [J]. CIESC Journal, 2023, 74(6): 2503-2521.
[6]	Cheng YUN, Qianlin WANG, Feng CHEN, Xin ZHANG, Zhan DOU, Tingjun YAN. Deep-mining risk evolution path of chemical processes based on community structure [J]. CIESC Journal, 2023, 74(4): 1639-1650.
[7]	Shumin ZHENG, Pengcheng GUO, Jianguo YAN, Shuai WANG, Wenbo LI, Qi ZHOU. Experimental and predictive study on pressure drop of subcooled flow boiling in a mini-channel [J]. CIESC Journal, 2023, 74(4): 1549-1560.
[8]	Xuerong GU, Shuoshi LIU, Siyu YANG. Research on multi-parameter optimization method based on parallel EGO and surrogate-assisted model [J]. CIESC Journal, 2023, 74(3): 1205-1215.
[9]	Yuanjing MAO, Zhi YANG, Songping MO, Hao GUO, Ying CHEN, Xianglong LUO, Jianyong CHEN, Yingzong LIANG. Estimation of SAFT-VR Mie equation of state parameters and thermodynamic properties of C₆—C₁₀ alcohols [J]. CIESC Journal, 2023, 74(3): 1033-1041.
[10]	Xinyuan WU, Qilei LIU, Boyuan CAO, Lei ZHANG, Jian DU. Group2vec: group vector representation and its property prediction applications based on unsupervised machine learning [J]. CIESC Journal, 2023, 74(3): 1187-1194.
[11]	Sheng’an ZHANG, Guilian LIU. Multi-objective optimization of high-efficiency solar water electrolysis hydrogen production system and its performance [J]. CIESC Journal, 2023, 74(3): 1260-1274.
[12]	Haiou YUAN, Fangjun YE, Shuo ZHANG, Yiqing LUO, Xigang YUAN. Synthesis of heat-integrated distillation sequences with intermediate heat exchangers [J]. CIESC Journal, 2023, 74(2): 796-806.
[13]	Jiahui CHEN, Xinze YANG, Guzhong CHEN, Zhen SONG, Zhiwen QI. A critical discussion on developing molecular property prediction models: density of ionic liquids as example [J]. CIESC Journal, 2023, 74(2): 630-641.
[14]	Xuejin GAO, Kun CHENG, Huayun HAN, Huihui Gao, Yongsheng QI. Fault diagnosis of chillers using central loss conditional generative adversarial network [J]. CIESC Journal, 2022, 73(9): 3950-3962.
[15]	Yalin WANG, Yuqing PAN, Chenliang LIU. Intermittent process monitoring based on GSA-LSTM dynamic structure feature extraction [J]. CIESC Journal, 2022, 73(9): 3994-4002.