基于ESO的动作依赖启发式动态规划控制及其在四级连续搅拌釜系统应用

doi:10.11949/0438-1157.20251185

摘要/Abstract

摘要：

本文针对化工过程中四级连续搅拌反应釜（CSTR）系统，提出了一种融合扩张状态观测器（ESO）与执行依赖启发式动态规划（ADHDP）的智能控制策略，旨在解决该系统面临强耦合和未知动态等情况下的温度控制问题。所提策略首先使用ESO实时估计和补偿子系统的“总扰动”，同时完成对子系统之间的解耦，随后针对被补偿的系统设计一种ADHDP算法，以获得最优虚拟控制律。本文所提方法做到了分布式抗扰和集中式控制，这不仅处理了系统强耦合问题，同时ESO的存在也减轻了ADHDP算法的抗扰压力。为验证所提方法的优越性，本文设计了三种对比算法，包括PID控制算法、自抗扰控制（ADRC）算法和ADHDP算法，结果表明本文所提算法在响应时间和稳态误差等方面均优于其他三种算法。

关键词: 自适应动态规划, ESO, 四级CSTR, 神经网络, 过程控制, 动态建模

Abstract:

An intelligent control strategy for a four-stage continuous stirred tank reactor (CSTR) system in chemical processes is proposed, which integrates an extended state observer (ESO) with action-dependent heuristic dynamic programming (ADHDP). The aim is to address the challenge of temperature control in the presence of strong coupling and unknown dynamics. First, the ESO is used to estimate and compensate the "total disturbance" of each subsystem in real time while achieving decoupling among the subsystems. Then, for the disturbance-compensated system, an ADHDP algorithm is designed to obtain the optimal virtual control law. The proposed method achieves decentralized disturbance rejection and centralized control, which not only addresses the system's strong coupling issues but also reduces the burden on the ADHDP algorithm thanks to the presence of the ESO. To verify the superiority of the proposed method, three comparative algorithms were designed in this study, including a PID control algorithm, an active disturbance rejection control (ADRC) algorithm, and an ADHDP algorithm. The results show that the algorithm proposed in this paper outperforms the other three algorithms in terms of response time, steady-state error, and other performance metrics.

Key words: ADP, ESO, four-stage CSTR process, neural networks, process control, dynamic modeling

中图分类号:

TP 273

陈郇, 李大字. 基于ESO的动作依赖启发式动态规划控制及其在四级连续搅拌釜系统应用[J]. 化工学报, DOI: 10.11949/0438-1157.20251185.

Xun CHEN, Dazi LI. ESO-based action dependent heuristic dynamic programming control and its' application on a four-stage CSTR process[J]. CIESC Journal, DOI: 10.11949/0438-1157.20251185.

图/表 10

图1 四级CSTR工艺流程图

Fig.1 A schematic of the four-stage CSTR process

表1 四级CSTR系统参数

Table 1 Parameters of the four-stage CSTR process

参数	单位		参数	单位	数值
$C A 01$	kmol/m³	4.0	$T 01$	K	300
$C A 02$	kmol/m³	2.0	$T 02$	K	300
$C A 03$	kmol/m³	3.0	$T 03$	K	300
$C A 04$	kmol/m³	3.5	$T 04$	K	300
$V 1$	m³	1	$Δ H 1$	kJ/mol	-5.0×10⁴
$V 2$	m³	3	$Δ H 2$	kJ/mol	-5.2×10⁴
$V 3$	m³	4	$Δ H 3$	kJ/mol	-5.0×10⁴
$V 4$	m³	6	$k 1$	h^-1	3.0×10⁶
$F 1$	m³/h	35	$k 2$	h^-1	3.0×10⁵
$F 2$	m³/h	45	$k 3$	h^-1	3.0×10⁵
$F 3$	m³/h	33	$c p$	kJ/(kg∙K)	0.231
$E 1$	kJ/kmol	5.0×10⁴	$ρ$	kg/m³	1000
$E 2$	kJ/kmol	7.5×10⁴	$F 01$	m³/h	5
$E 3$	kJ/kmol	7.53×10⁴	$F 02$	m³/h	10
$F r 1$	m³/h	20	$F 03$	m³/h	8
$F r 2$	m³/h	10	$F 04$	m³/h	12
$R$	kJ/(mol∙K)	8.314

表1 四级CSTR系统参数

Table 1 Parameters of the four-stage CSTR process

参数	单位		参数	单位	数值
$C A 01$	kmol/m³	4.0	$T 01$	K	300
$C A 02$	kmol/m³	2.0	$T 02$	K	300
$C A 03$	kmol/m³	3.0	$T 03$	K	300
$C A 04$	kmol/m³	3.5	$T 04$	K	300
$V 1$	m³	1	$Δ H 1$	kJ/mol	-5.0×10⁴
$V 2$	m³	3	$Δ H 2$	kJ/mol	-5.2×10⁴
$V 3$	m³	4	$Δ H 3$	kJ/mol	-5.0×10⁴
$V 4$	m³	6	$k 1$	h^-1	3.0×10⁶
$F 1$	m³/h	35	$k 2$	h^-1	3.0×10⁵
$F 2$	m³/h	45	$k 3$	h^-1	3.0×10⁵
$F 3$	m³/h	33	$c p$	kJ/(kg∙K)	0.231
$E 1$	kJ/kmol	5.0×10⁴	$ρ$	kg/m³	1000
$E 2$	kJ/kmol	7.5×10⁴	$F 01$	m³/h	5
$E 3$	kJ/kmol	7.53×10⁴	$F 02$	m³/h	10
$F r 1$	m³/h	20	$F 03$	m³/h	8
$F r 2$	m³/h	10	$F 04$	m³/h	12
$R$	kJ/(mol∙K)	8.314

图2 ESO-ADHDP结构框图

Fig.2 ESO-ADHDP structure diagram

表2 四级CSTR系统ESO-ADHDP算法流程

Table 2 Four-stage CSTR ESO-ADHDP Algorithm Process

四级CSTR系统ESO-ADHDP算法流程
随机初始化评价网络权重矩阵 $W c h o$ 和 $W c i h$ ，随机初始化执行网络权重矩阵 $W a h o$ 和 $W a i h$ ；
初始化系统状态 $X 0$ ，提取温度初始化状态向量 $T 0$ ；
初始化四个子系统的ESO状态 $x ̂ i 1$ 和 $x ̂ i 2$ ， $i = 1,2, 3,4$ ；
for $k = 1 → t s p a n / T s$ ；
计算四个子系统CSTR_i的温度 $T i$ 与设定值 $R T i$ 的误差，得到温度误差向量 $e T k$ ；
通过执行网络计算 $Q 0 k$ ： $Q 0 k ← a c t o r N N e T k, W a i h k, W a h o k$ ；
计算 $Q k$ ，其中 $Q i k ← Q 0 i k - x ̂ i 2 k / b ̂ 0 i$ ；
使用 $Q k$ 与四级CSTR系统交互获得下一个时刻状态： $X k + 1 ← C S T R p l a n t X k, Q k$ ；
通过公式(20)计算ESO下一时刻估计状态的采样值 $x ̂ i 1 k + 1$ 和 $x ̂ i 2 k + 1$ ， $i = 1,2, 3,4$ ；
提取温度状态向量 $T k + 1$ ，温度误差向量 $e T k + 1$ ；
通过执行网络计算 $Q 0 k + 1$ ： $Q 0 k + 1 ← a c t o r N N e T k + 1, W a i h k, W a h o k$ ；
计算相邻两个时刻的 $J ̂$ : $J ̂ k ← c r i t i c N N Q 0 k, e T k, W c i h k, W c h o k$ ; $J ̂ k + 1 ← c r i t i c N N Q 0 k + 1, e T k + 1, W c i h k, W c h o k$ ;
计算TD误差： $U f k + 1 ← e T T k Q e T k + Q 0 T k R Q 0 k$ ; $δ k ← J ̂ k - U f k + 1 - γ J ̂ k + 1$ ;
根据公式(28)和(31)更新评价网络权重矩阵： $W c h o k + 1 ← W c h o k + Δ W c h o k;$ $W c i h k + 1 ← W c i h k + Δ W c i h k;$
根据公式(37)和(40)更新执行网络权重矩阵： $W a h o k + 1 ← W a h o k + Δ W a h o k;$ $W a i h k + 1 ← W a i h k + Δ W a i h k;$
end for

表2 四级CSTR系统ESO-ADHDP算法流程

Table 2 Four-stage CSTR ESO-ADHDP Algorithm Process

四级CSTR系统ESO-ADHDP算法流程
随机初始化评价网络权重矩阵 $W c h o$ 和 $W c i h$ ，随机初始化执行网络权重矩阵 $W a h o$ 和 $W a i h$ ；
初始化系统状态 $X 0$ ，提取温度初始化状态向量 $T 0$ ；
初始化四个子系统的ESO状态 $x ̂ i 1$ 和 $x ̂ i 2$ ， $i = 1,2, 3,4$ ；
for $k = 1 → t s p a n / T s$ ；
计算四个子系统CSTR_i的温度 $T i$ 与设定值 $R T i$ 的误差，得到温度误差向量 $e T k$ ；
通过执行网络计算 $Q 0 k$ ： $Q 0 k ← a c t o r N N e T k, W a i h k, W a h o k$ ；
计算 $Q k$ ，其中 $Q i k ← Q 0 i k - x ̂ i 2 k / b ̂ 0 i$ ；
使用 $Q k$ 与四级CSTR系统交互获得下一个时刻状态： $X k + 1 ← C S T R p l a n t X k, Q k$ ；
通过公式(20)计算ESO下一时刻估计状态的采样值 $x ̂ i 1 k + 1$ 和 $x ̂ i 2 k + 1$ ， $i = 1,2, 3,4$ ；
提取温度状态向量 $T k + 1$ ，温度误差向量 $e T k + 1$ ；
通过执行网络计算 $Q 0 k + 1$ ： $Q 0 k + 1 ← a c t o r N N e T k + 1, W a i h k, W a h o k$ ；
计算相邻两个时刻的 $J ̂$ : $J ̂ k ← c r i t i c N N Q 0 k, e T k, W c i h k, W c h o k$ ; $J ̂ k + 1 ← c r i t i c N N Q 0 k + 1, e T k + 1, W c i h k, W c h o k$ ;
计算TD误差： $U f k + 1 ← e T T k Q e T k + Q 0 T k R Q 0 k$ ; $δ k ← J ̂ k - U f k + 1 - γ J ̂ k + 1$ ;
根据公式(28)和(31)更新评价网络权重矩阵： $W c h o k + 1 ← W c h o k + Δ W c h o k;$ $W c i h k + 1 ← W c i h k + Δ W c i h k;$
根据公式(37)和(40)更新执行网络权重矩阵： $W a h o k + 1 ← W a h o k + Δ W a h o k;$ $W a i h k + 1 ← W a i h k + Δ W a i h k;$
end for

表3 不同控制策略参数

Table 3 Parameters of the four controllers

控制策略	参数	值		控制策略	参数
PID	$k p 1$	1500	ADRC	$k p 1$	1000
	$k p 2$	2000		$k p 2$	1000
	$k p 3$	2400		$k p 3$	1000
	$k p 4$	2800		$k p 4$	1000
	$k i 1$	10		$ω o 1$	100
	$k i 2$	10		$ω o 2$	100
	$k i 3$	10		$ω o 3$	100
	$k i 4$	10		$ω o 4$	100
ESO-ADHDP	$l c$	0.01	ADHDP	$l c$	0.01
	$l a$	0.01		$l a$	0.01
	$γ$	0.9		$γ$	0.9
	$ω o 1$	100		$Q$	$e y e 4$
	$ω o 2$	100		$R$	$10 - 8 × e y e 4$
	$ω o 3$	100
	$ω o 4$	100
	$Q$	$e y e 4$
	$R$	$10 - 8 × e y e 4$

表3 不同控制策略参数

Table 3 Parameters of the four controllers

控制策略	参数	值		控制策略	参数
PID	$k p 1$	1500	ADRC	$k p 1$	1000
	$k p 2$	2000		$k p 2$	1000
	$k p 3$	2400		$k p 3$	1000
	$k p 4$	2800		$k p 4$	1000
	$k i 1$	10		$ω o 1$	100
	$k i 2$	10		$ω o 2$	100
	$k i 3$	10		$ω o 3$	100
	$k i 4$	10		$ω o 4$	100
ESO-ADHDP	$l c$	0.01	ADHDP	$l c$	0.01
	$l a$	0.01		$l a$	0.01
	$γ$	0.9		$γ$	0.9
	$ω o 1$	100		$Q$	$e y e 4$
	$ω o 2$	100		$R$	$10 - 8 × e y e 4$
	$ω o 3$	100
	$ω o 4$	100
	$Q$	$e y e 4$
	$R$	$10 - 8 × e y e 4$

图3 四级CSTR出料温度曲线

Fig.3 Temperature curve of the four-stage CSTR

图4 虚拟控制量Q0和总扰动估计值f^曲线

Fig.4 Curve of virtual control signal Q0 and total disturbance estimation f^

图5 ESO-ADHDP策略的TD误差收敛曲线

Fig.5 TD Error Convergence curve of the ESO-ADHDP strategy

图6 ESO-ADHDP控制策略的损失函数收敛曲线

Fig.6 Convergence curve of the loss function for the ESO-ADHDP control strategy

图7 不同CSTR出料浓度曲线

Fig.7 Concentration curve of the four CSTRs

参考文献 33

[1]	Hu C T, Finkelstein J E, Wu W, et al. Development of an automated multi-stage continuous reactive crystallization system with in-line PATs for high viscosity process[J]. Reaction Chemistry & Engineering, 2018, 3(5): 658-667.
[2]	钟子豪, 刘塞尔, 商敏静, 等. 基于磁力搅拌的微型CSTR流动特征及微观混合性能[J]. 化工学报, 2024, 75(11): 4217-4225.
	Zhong Z H, Liu S E, Shang M J, et al. Flow characteristics and micromixing performance of micro-CSTR with magnetic stirring[J]. CIESC Journal, 2024, 75(11): 4217-4225.
[3]	Navid Y, Mehdi S, Masoud A, et al. Design of Nonlinear CSTR Control System using Active Disturbance Rejection Control Optimized by Asexual Reproduction Optimization[J]. Journal of Automation and Control, 2015, 3(2): 36-42.
[4]	王立敏, 杨继胜, 于晶贤, 等. 基于T-S模糊模型的间歇过程的迭代学习容错控制[J]. 化工学报, 2017, 68(3): 1081-1089.
	Wang L M, Yang J S, Yu J X, et al. Iterative learning fault-tolerant control for batch processes based on T-S fuzzy model[J]. CIESC Journal, 2017, 68(3): 1081-1089.
[5]	刘凯, 辛丽平, 刘家硕, 等. 连续搅拌反应釜的固定时间命令滤波跟踪控制[J]. 控制与决策, 2024, 39(6): 1936-1942.
	Liu K, Xin L P, Liu J S, et al. Fixed-time command filter tracking control of continuous stirred tank reactor[J]. Control and Decision, 2024, 39(6): 1936-1942.
[6]	Li D Z, Li Z, Gao Z Q, et al. Active disturbance rejection-based high-precision temperature control of a semibatch emulsion polymerization reactor[J]. Industrial & Engineering Chemistry Research, 2014, 53(8): 3210-3221.
[7]	Lee J Y, Jin G G, So G B, et al. Adaptive nonlinear proportional–integral–derivative control of a continuous stirred tank reactor process using a radial basis function neural network[J]. Algorithms, 2025, 18(7): 442.
[8]	Siddiqui M A, Anwar M N, Laskar S H. Control of nonlinear jacketed continuous stirred tank reactor using different control structures[J]. Journal of Process Control, 2021, 108: 112-124.
[9]	Cherkasov N, Adams S J, Bainbridge E G A, et al. Continuous stirred tank reactors in fine chemical synthesis for efficient mixing, solids-handling, and rapid scale-up[J]. Reaction Chemistry & Engineering, 2023, 8(2): 266-277.
[10]	周红标, 张钰, 柏小颖, 等. 基于自适应模糊神经网络的非线性系统模型预测控制[J]. 化工学报, 2020, 71(7): 3201-3212.
	Zhou H B, Zhang Y, Bai X Y, et al. Model predictive control of nonlinear system based on adaptive fuzzy neural network[J]. CIESC Journal, 2020, 71(7): 3201-3212.
[11]	孔晓涵, 辛丽平, 柴欣生. 串级连续搅拌反应釜的有限时间命令滤波控制[J]. 控制与决策, 2022, 37(9): 2245-2254.
	Kong X H, Xin L P, Chai X S. Finite-time command filter control of cascade continuous stirred tank reactors[J]. Control and Decision, 2022, 37(9): 2245-2254.
[12]	Chen S, Wu Z, Rincon D, et al. Machine learning-based distributed model predictive control of nonlinear processes[J]. AIChE Journal, 2020, 66(11): e17013.
[13]	Liu Y F, Ma M, Liu Y F, et al. Finite-time fuzzy tracking control for two-stage continuous stirred tank reactor: a gradient descent approach via armijo line search[J]. Electronics, 2025, 14(20): 4069.
[14]	Zhang F, Wang L. Disturbance rejection design for Gaussian process-based model predictive control using extended state observer[J]. Computers & Chemical Engineering,2024,186: 108708.
[15]	Xiao M, Zeng M, Li Y, et al. Drive control of proportional valves based on active disturbance rejection control[J]. Engineering Research Express, 2025, 7(2): 025012.
[16]	Ye Y H, Cheng Y, Zhou F, et al. Optimization of active disturbance rejection controller for distillation process based on quantitative feedback theory[J]. Processes, 2025, 13(5): 1436.
[17]	Song X D, Zhao Y D, Li Z H, et al. A dual-loop modified active disturbance rejection control scheme for a high-purity distillation column[J]. Processes, 2025, 13(5): 1359.
[18]	Peng L H, Xiong X Y, Wang L J. Research on yarn tension control technology for knitting underwear machine based on adaptive ADRC[J]. Scientific Reports, 2025, 15: 9750.
[19]	程赟, 陈增强, 孙明玮, 等. 多变量逆解耦自抗扰控制及其在精馏塔过程中的应用[J]. 自动化学报, 2017, 43(6): 1080-1088.
	Cheng Y, Chen Z Q, Sun M W, et al. Multivariable inverted decoupling active disturbance rejection control and its application to a distillation column process[J]. Acta Automatica Sinica, 2017, 43(6): 1080-1088.
[20]	Liu R J, Nie Z Y, Shao H, et al. Active disturbance rejection control for non-minimum phase systems under plant reconstruction[J]. ISA Transactions, 2023, 134: 497-510.
[21]	郭晓临, 刘洋, 林娜, 等. 基于扩展状态观测器的量化无模型自适应迭代学习控制[J]. 控制理论与应用, 2025, 42(2): 253-262.
	Guo X L, Liu Y, Lin N, et al. Extended state observer-based quantitative model-free adaptive iterative learning control[J]. Control Theory & Applications, 2025, 42(2): 253-262.
[22]	Li D Z, Wang Z, Yu W L, et al. Application of LADRC with stability region for a hydrotreating back-flushing process[J]. Control Engineering Practice, 2018, 79: 185-194.
[23]	Li D Z, Meng N J, Song T H. Learning control of fermentation process with an improved DHP algorithm[J]. Chinese Journal of Chemical Engineering, 2016, 24(10): 1399-1405.
[24]	Li A, Shen Y H, Du B, et al. Backstepping-based finite-horizon optimization for pitching attitude control of aircraft[J]. Aerospace, 2025, 12(8): 653.
[25]	谭旭峰, 李媛, 刘洋. 基于自适应动态规划的随机时滞线性二次型最优跟踪控制[J]. 系统科学与数学, 2024, 44(1): 17-30.
	Tan X F, Li Y, Liu Y. Stochastic linear quadratic optimal tracking control with time-delays based on adaptive dynamic programming[J]. Journal of Systems Science and Mathematical Sciences, 2024, 44(1): 17-30.
[26]	Wang D, Ren J, Huang H M, et al. Particle swarm optimization for adaptive-critic feedback control with power system applications[J]. Chinese Journal of Electronics, 2025, 34(4): 1265-1274.
[27]	Li L T, Li D Z, Song T H, et al. Actor–critic learning control with regularization and feature selection in policy gradient estimation[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021, 32(3): 1217-1227.
[28]	Luo S J, Xue K Z W, . An ADP-based robust control scheme for nonaffine nonlinear systems with uncertainties and input constraints[J]. Chinese Physics B, 2025, 34(6): 251-260.
[29]	Vu M T, Kim S H, Pham D H, et al. Adaptive dynamic programming-based intelligent finite-time flexible SMC for stabilizing fractional-order four-wing chaotic systems[J]. Mathematics, 2025, 13(13): 2078.
[30]	张化光, 张欣, 罗艳红, 等. 自适应动态规划综述[J]. 自动化学报, 2013, 39(4): 303-311.
	Zhang H G, Zhang X, Luo Y H, et al. An overview of research on adaptive dynamic programming[J]. Acta Automatica Sinica, 2013, 39(4): 303-311.
[31]	王鼎, 赵明明, 刘德荣, 等. 数据驱动自适应评判控制研究进展[J]. 自动化学报, 2025, 51(6): 1170-1190.
	Wang D, Zhao M M, Liu D R, et al. Research advances on data-driven adaptive critic control[J]. Acta Automatica Sinica, 2025, 51(6): 1170-1190.
[32]	Liu S Y, Yin X Y, Liu J B, et al. Distributed simultaneous state and parameter estimation of nonlinear systems[J]. Chemical Engineering Research and Design, 2022, 181: 74-86.
[33]	Sokolov Y, Kozma R, Werbos L D, et al. Complete stability analysis of a heuristic approximate dynamic programming control design[J]. Automatica, 2015, 59: 9-18.