Quantile regression CGAN based virtual samples generation and its applications to process modeling

doi:10.11949/0438-1157.20201748

Abstract

Abstract:

Complex industry processes suffer from the problems on data scarcity of training samples which are collected for modeling, as a result of inaccessibility of difficult-to-measure variables or high cost in time or economy. To tackle the issues, we proposed a novel virtual sample generation embedding quantile regression into conditional generative adversarial networks (QRCGAN). First of all, a regression is embedded in the standard CGAN “generator-discriminator” two-element game structure, so that the model not only has the ability to generate label samples, but also has the ability to handle regression prediction problems. Secondly, the regressor is implemented by the quantile regression neural network (QRNN), together with the discriminator and generator for simultaneous adversarial training. Once the model reaches the Nash equilibrium, with the help of the QRNN regressor, the generator can generate new samples that fall within a certain confidence interval. Moreover, the Kullback-Leibler (KL) divergence was used to evaluate the quality of the generated samples. Finally, the effectiveness of the proposed method is verified by standard function data and actual chemical process data.

Key words: virtual sample generation, CGAN, quantile regression, data scarcity, soft sensing, deep learning

摘要：

针对复杂工业过程因难以检测变量或因时间上和经济上成本因素导致的建模样本稀缺问题，提出了一种将分位数回归（quantile regression）嵌入到条件生成式对抗网络（conditional generative adversarial network， CGAN）的虚拟样本生成方法QRCGAN。首先，在标准CGAN“生成器-判别器”两元对弈结构中嵌入回归器，使模型不仅具备标签样本生成能力，同时也具备处理回归预测问题的能力。其次，以分位数回归神经网络实现回归器，连同判别器和生成器进行同步对抗训练。当模型到达Nash平衡时，在分位数回归神经网络回归器的作用下，生成器能够产生落在一定置信区间的新样本。然后，利用Kullback-Leibler（KL）散度评估生成样本的质量。最后，通过标准函数数据和实际化工过程数据验证所提方法的有效性。

关键词: 虚拟样本生成, CGAN, 分位数回归, 数据稀缺, 软测量, 深度学习

CLC Number:

CHEN Zhongsheng, ZHU Meiyu, HE Yanlin, XU Yuan, ZHU Qunxiong. Quantile regression CGAN based virtual samples generation and its applications to process modeling[J]. CIESC Journal, 2021, 72(3): 1529-1538.

陈忠圣, 朱梅玉, 贺彦林, 徐圆, 朱群雄. 基于分位数回归CGAN的虚拟样本生成方法及其过程建模应用[J]. 化工学报, 2021, 72(3): 1529-1538.

Figures/Tables 9

Fig.1 Schematic diagram of the proposed QRCGAN

Table 1 Benchmarking functions used and its definitions

记号	基准函数	噪声分布	定义区间
LinHomo	y=x + ε	ε ~ N (0,0.05²)	x $∈$ [0, 1]
LinHetero	y=x + ε	ε ~ N (0,0.01x²)	x $∈$ [0, 1]
SinOsc	y=x + 0.2sin(20x) +ε	ε ~ N (0,0.05²)	x $∈$ [0, 1]

Table 1 Benchmarking functions used and its definitions

记号	基准函数	噪声分布	定义区间
LinHomo	y=x + ε	ε ~ N (0,0.05²)	x $∈$ [0, 1]
LinHetero	y=x + ε	ε ~ N (0,0.01x²)	x $∈$ [0, 1]
SinOsc	y=x + 0.2sin(20x) +ε	ε ~ N (0,0.05²)	x $∈$ [0, 1]

Fig.2 Conditional density estimation for the benchmark function LinHomo when given different y

Fig.3 Conditional density estimation for the benchmark function LinHetero when given different y

Fig.4 Conditional density estimation for the benchmark function SinOsc when given different y

Fig.5 KL divergence against the updates of generator

Fig.6 Cloud points of virtual samples

Table 2 Quality evaluation for cloud points of virtual samples

模型	LinHomo	LinHetero	SinOsc
QRCGAN	0.418	1.315	3.028
CGAN	0.611	1.932	8.328

Fig.7 Comparison of predictions of models

References 22

1	Zhu J, Ge Z, Song Z, et al. Review and big data perspectives on robust data mining approaches for industrial process modeling with outliers and missing data[J]. Annual Reviews in Control, 2018, 46: 107-133.
2	朱群雄, 耿志强, 徐圆, 等. 数据和知识融合驱动的智能过程系统工程研究进展[J]. 北京化工大学学报(自然科学版), 2018, 45(5): 143-152.
	Zhu Q X, Geng Z Q, Xu Y, et al. Progress in intelligent process system engineering driven by data and knowledge fusion[J]. Journal of Beijing University of Chemical Technology (Natural Science Edition), 2018, 45(5): 143-152.
3	Zhu Q X, Zhang X H, He Y L. Novel virtual sample generation based on locally linear embedding for optimizing the small sample problem: case of soft sensor applications[J]. Industrial & Engineering Chemistry Research, 2020, 59(40): 17977-17986.
4	Zhu Q X, Chen Z S, Zhang X H, et al. Dealing with small sample size problems in process industry using virtual sample generation: a Kriging-based approach[J]. Soft Computing, 2020, 24(9): 6889-6902.
5	乔俊飞, 郭子豪, 汤健. 基于改进大趋势扩散和隐含层插值的虚拟样本生成方法及应用[J]. 化工学报, 2020, 71(12): 5681-5695.
	Qiao J F, Guo Z H, Tang J. Virtual sample generation method based on improved megatrend diffusion and hidden layer interpolation with its application[J]. CIESC Journal, 2020, 71(12): 5681-5695.
6	Zhu Q X, Gong H F, Xu Y, et al. A bootstrap based virtual sample generation method for improving the accuracy of modeling complex chemical processes using small datasets[C]// 6th Data Driven Control and Learning Systems Conference. Chongqing, China, 2017.
7	Coqueret G. Approximate NORTA simulations for virtual sample generation[J]. Expert Systems with Applications, 2017, 73: 69-81.
8	Li D C, Wu C S, Tsai T I, et al. Using mega-trend-diffusion and artificial samples in small data set learning for early flexible manufacturing system scheduling knowledge[J]. Computers & Operations Research, 2007, 34(4): 966-982.
9	朱宝, 陈忠圣, 余乐安. 一种新颖的小样本整体趋势扩散技术[J]. 化工学报, 2016, 67(3): 820-826.
	Zhu B, Chen Z S, Yu L A. A novel mega-trend-diffusion for small sample[J]. CIESC Journal, 2016, 67(3): 820-826.
10	Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial networks[J]. arXiv, 2014, arXiv: 1406.2661 [stat.ML]. .
11	Kingma D P, Welling M. Auto-encoding variational bayes[J]. arXiv, 2013, arXiv: 1312.6114 [stat.ML]. .
12	Salimans T, Goodfellow I, Zaremba W, et al. Improved techniques for training GANs[J]. arXiv, 2016, arXiv: 1606.03498 [cs.LG]. .
13	Mirza M, Osindero S. Conditional generative adversarial nets[J]. arXiv, 2014, arXiv: 1411.1784 [cs.LG]. .
14	Radford A, Metz L, Chintala S. Unsupervised representation learning with deep convolutional generative adversarial networks[J]. arXiv, 2016, arXiv: 1511.06434 [cs.LG]. .
15	Arjovsky M, Chintala S, Bottou L. Wasserstein GAN[J]. arXiv, 2017, arXiv: 1701.07875 [stat.ML]. .
16	Zhu J, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[J]. arXiv, 2017, arXiv: 1703.10593 [cs.CV]. .
17	Donahue J, Krähenbühl P, Darrell T. Adversarial feature learning[J]. arXiv, 2016, arXiv: 1605.09782 [cs.LG]. .
18	Rezagholiradeh M, Haidar M A. Reg-Gan: semi-supervised learning based on generative adversarial networks for regression [C]// International Conference on Acoustics, Speech and Signal Processing. Calgary, Canada, 2018.
19	庞昊, 高金峰, 杜耀恒. 基于时间卷积网络分位数回归的短期负荷概率密度预测方法[J]. 电网技术, 2020, 44(4): 1343-1350.
	Pang H, Gao J F, Du Y H. A short-term load probability density prediction based on quantile regression of time convolution network[J]. Power System Technology, 2020, 44(4): 1343-1350.
20	Taylor J W. A quantile regression approach to estimating the distribution of multiperiod returns[J]. The Journal of Derivatives, 1999, 7(1): 64-78.
21	Boget Y. Adversarial regression. Generative adversarial networks for non-linear regression: theory and assessment[D]. Switzerland: University of Neuchâtel, 2019.
22	朱群雄, 张晓晗, 顾祥柏, 等. 基于特征提取的函数连接神经网络研究及其化工过程建模应用[J]. 化工学报, 2018, 69(3): 907-912.
	Zhu Q X, Zhang X H, Gu X B, et al. Research and application of feature extraction derived functional link neural network[J]. CIESC Journal, 2018, 69(3): 907-912.

[1]	Kaijie WEN, Li GUO, Zhaojie XIA, Jianhua CHEN. A rapid simulation method of gas-solid flow by coupling CFD and deep learning [J]. CIESC Journal, 2023, 74(9): 3775-3785.
[2]	Shunhua LUO, Zhenlei WANG, Xin WANG. Semi-supervised soft sensor modeling based on two-subspace co-training algorithm [J]. CIESC Journal, 2022, 73(3): 1270-1279.
[3]	Shuping QI, Wenlong WANG, Lei ZHANG, Jian DU. A deep learning-based model for predicting the stability constants of metal ions with organic ligands [J]. CIESC Journal, 2022, 73(12): 5461-5468.
[4]	XIE Haoyuan, HUANG Qunxing, LIN Xiaoqing, LI Xiaodong, YAN Jianhua. Study on the calorific value prediction of municipal solid wastes by image deep learning [J]. CIESC Journal, 2021, 72(5): 2773-2782.
[5]	YU Chengyuan, WU Jinkui, ZHOU Li, JI Xu, DAI Yiyang, DANG Yagu. Prediction of energy conversion efficiency of organic solar cells based on deep learning [J]. CIESC Journal, 2021, 72(3): 1487-1495.
[6]	Xiaohui WANG, Yanjiang WANG, Xiaogang DENG, Zheng ZHANG. Industrial process fault detection using weighted deep support vector data description [J]. CIESC Journal, 2021, 72(11): 5707-5716.
[7]	Linzi YIN, Yuyin GUAN, Zhaohui JIANG, Xuemei XU. Optimal method of selecting silicon content data in blast furnace hot metal based on k-means++ [J]. CIESC Journal, 2020, 71(8): 3661-3670.
[8]	QIAO Junfei,GUO Zihao,TANG Jian. Virtual sample generation method based on improved megatrend diffusion and hidden layer interpolation with its application [J]. CIESC Journal, 2020, 71(12): 5681-5695.
[9]	Luyao TIAN, Zihao WANG, Yang SU, Huaqiang WEN, Weifeng SHEN. Research advances in deep learning based quantitative structure-property relationship modeling of solvents [J]. CIESC Journal, 2020, 71(10): 4462-4472.
[10]	Hengchang GU, Peng MU, Jianwei LI. Modeling and application of ethylene cracking furnace based on cross-iterative BLSTM network [J]. CIESC Journal, 2019, 70(2): 548-555.
[11]	DONG Shun, LI Yiguo, SUN Shuanzhu, LIU Xichui, SHEN Jiong. Fault detection method based on state space-PCANet [J]. CIESC Journal, 2018, 69(8): 3528-3536.
[12]	WANG Kangcheng, SHANG Chao, KE Wensi, JIANG Yongheng, HUANG Dexian. Automatic structure and parameters tuning method for deep neural network soft sensor in chemical industries [J]. CIESC Journal, 2018, 69(3): 900-906.
[13]	ZHU Xianglin, LING Jing, WANG Bo, HAO Jianhua, DING Yuhan. Soft-sensing modeling of marine protease fermentation process based on improved PSO-RBFNN [J]. CIESC Journal, 2018, 69(3): 1221-1227.
[14]	WANG Gongming, LI Wenjing, QIAO Junfei. Prediction of effluent total phosphorus using PLSR-based adaptive deep belief network [J]. CIESC Journal, 2017, 68(5): 1987-1997.
[15]	XU Wenxing, HE Qian, DAI Bo, ZHANG Huiping. Adaptive hybrid particle swarm optimizer for constrained optimization problems and its application in parameter estimation of soft sensing [J]. CIESC Journal, 2015, 66(1): 222-227.