Optimal control for neutralization process of citric acid through tricalcium reaction based on reinforcement learning algorithm

doi:10.11949/0438-1157.20241249

Abstract

Abstract:

Tricalcium neutralization process is a crucial procedure in the citric acid extraction technology and a key stage influencing the quality and yield of the final citric acid products. The process is characterized by time delay, absence of reference trajectory, significant variations in initial materials, irreversible reaction, which is difficult to be optimally controlled by traditional control algorithms. Aiming at the above problems, the actual tricalcium citrate neutralization process is optimized and controlled by the reinforcement learning algorithm deep deterministic policy (DDPG). Considering that model-based reinforcement learning approach enables the agent to conduct cost-free exploration within the learned model, the long short term memory (LSTM) model of the tricalcium neutralization process is established, and its loss function is improved to reduce the gap between the simulation model and the actual environment. Subsequently, the model is used to participate in reinforcement learning training, and finally the trained control strategy is used in the actual tricalcium neutralization process. The experimental results indicate that this method can successfully apply the optimal strategy trained through simulation to the actual tricalcium neutralization process, achieving satisfactory results.

Key words: tricalcium neutralization process, optimal control, DDPG, model-based reinforcement learning, LSTM

摘要：

三钙中和过程是柠檬酸提取工艺的重要工序，是影响柠檬酸成品质量、产品收率的关键工段。该过程具有时滞、无参考轨迹、初始物料变化大、反应不可逆等特点，传统控制算法很难对其进行优化控制。针对上述问题，用强化学习算法深度确定性策略（DDPG）对实际的三钙中和过程进行优化控制。考虑到基于模型的强化学习方法可使智能体在学习的模型中进行无成本的探索，建立三钙中和过程的长短期记忆（LSTM）模型，并对其损失函数进行改进，减小了仿真模型与实际环境的差距，然后利用该模型进行强化学习训练，并将训练好的控制策略用于实际三钙中和过程。实验结果表明，该方法可以将仿真训练出的最优策略成功应用于实际三钙中和过程，并取得较好的结果。

关键词: 三钙中和过程, 优化控制, 深度确定性策略, 基于模型的强化学习, 长短期记忆

CLC Number:

TP 29

Lina ZHU, Maodong MIAO, Sai JIN, Zhonggai ZHAO, Fuxin SUN, Guiyang SHI, Fei LIU. Optimal control for neutralization process of citric acid through tricalcium reaction based on reinforcement learning algorithm[J]. CIESC Journal, 2025, 76(6): 2838-2847.

祝丽娜, 苗茂栋, 金赛, 赵忠盖, 孙福新, 石贵阳, 刘飞. 柠檬酸三钙中和过程的强化学习优化控制[J]. 化工学报, 2025, 76(6): 2838-2847.

Figures/Tables 11

Fig.1 Tricalcium reaction process of citric acid

Table 1 Earnings for different endpoint pH

组别	碳酸钙添加量/g	平衡时pH	混酸收益/ （CNY·m^-3）	年收益/ (10^-4 CNY)	对应收率/%
1	20.36	4.91	0	0	93.8
2	21.50	5.00	0.24	47	94.1
3	23.28	5.09	0.52	103	94.6
4	27.97	5.31	-0.25	-50	94.9
5	46.30	5.49	-5.27	-1044	95.3
6	66.99	5.70	-11.17	-2212	95.5

Fig.2 Structure of LSTM

Fig.3 Data preprocessing

Fig.4 LSTM prediction results

Fig.5 Block diagram of DDPG algorithm

Table 2 LSTM-DDPG algorithm

算法1:LSTM-DDPG
LSTM建模：
收集实际数据： $S T, t, A T, t, S T, t + 1$
记 $L S T M S T, t, A T, t m = S S, t + 1$
网络参数 $m = a r g m i n 1 N ∑ i N S S, i - S T, i 2$
DDPG：
随机初始化Critic网络 $Q S S, a S θ μ$ 和Actor网络 $μ S S θ μ$ ；初始化目标网络 $Q'$ 和 $μ'$ ： $θ Q' ← θ Q$ ， $θ μ' ← θ μ$ ；初始化经验回放池R
for episode = 1,M do:
行动探索，初始化随机噪声 $N t$ 和初始状态 $S S, 1$
for t = 1,T do:
$a S, t = μ S S, t θ μ + N t; S S, t + 1 = L S T M S S, t, A t$
执行动作 $a S, t$ ，得到奖励 $r S, t$ 和环境状态 $S S, t + 1$ 数据 $S S, t, a S, t, r S, t, S S, t + 1$ 存入R
从R中随机采样N组多维数组 $S S, i, a S, i, r S, i, S S, i + 1$
$y i = r i + γ Q' S S, i + 1, μ' S S, i + 1 θ μ' θ Q'$
最小化损失函数Loss来更新Critic网络
$L o s s = 1 N ∑ i N y i - Q S S, i + 1, a i θ Q 2$
采集策略梯度更新Actor策略网络
$∇ θ μ J θ μ = 1 N ∑ i N ∇ θ μ μ S S, i ∇ a Q S S, i, a θ Q a = μ S S, i$
更新目标网络
$θ Q' ← τ θ Q + 1 - τ θ Q' θ μ' ← τ θ μ + 1 - τ θ μ'$
end for
end for

Table 2 LSTM-DDPG algorithm

算法1:LSTM-DDPG
LSTM建模：
收集实际数据： $S T, t, A T, t, S T, t + 1$
记 $L S T M S T, t, A T, t m = S S, t + 1$
网络参数 $m = a r g m i n 1 N ∑ i N S S, i - S T, i 2$
DDPG：
随机初始化Critic网络 $Q S S, a S θ μ$ 和Actor网络 $μ S S θ μ$ ；初始化目标网络 $Q'$ 和 $μ'$ ： $θ Q' ← θ Q$ ， $θ μ' ← θ μ$ ；初始化经验回放池R
for episode = 1,M do:
行动探索，初始化随机噪声 $N t$ 和初始状态 $S S, 1$
for t = 1,T do:
$a S, t = μ S S, t θ μ + N t; S S, t + 1 = L S T M S S, t, A t$
执行动作 $a S, t$ ，得到奖励 $r S, t$ 和环境状态 $S S, t + 1$ 数据 $S S, t, a S, t, r S, t, S S, t + 1$ 存入R
从R中随机采样N组多维数组 $S S, i, a S, i, r S, i, S S, i + 1$
$y i = r i + γ Q' S S, i + 1, μ' S S, i + 1 θ μ' θ Q'$
最小化损失函数Loss来更新Critic网络
$L o s s = 1 N ∑ i N y i - Q S S, i + 1, a i θ Q 2$
采集策略梯度更新Actor策略网络
$∇ θ μ J θ μ = 1 N ∑ i N ∇ θ μ μ S S, i ∇ a Q S S, i, a θ Q a = μ S S, i$
更新目标网络
$θ Q' ← τ θ Q + 1 - τ θ Q' θ μ' ← τ θ μ + 1 - τ θ μ'$
end for
end for

Table 3 Network structure and parameters

参数		数值
LSTM网络	输入维度	2
	输出维度	1
	神经网络层数	64
	隐藏层层数	1
	时序长度	20
Actor网络	Linear+relu	16
Actor网络	Linear+tanh	16
Critic网络	Linear+relu	16
	Linear+relu	16
	Linear	16
Actor网络学习率		0.001
Critic网络学习率		0.002
奖励折扣系数		0.9
软更新系数		0.01
经验回放池容量		5000
小批量取样		150

Fig.6 Simulation results of LSTM-DDPG

Fig.7 Performance of simulation and practical applications

Table 4 Product quality at different initial pH

初始pH	终点pH	滤液滴定酸度/%	三钙水分/%	三钙RCS	碳酸钙过量比例
2.46	5.03	0.10	41.3	2.57	1.06
2.52	5.05	0.11	42.1	2.43	1.12
2.57	5.01	0.10	41.9	2.06	1.04
2.73	5.06	0.10	38.2	1.72	1.07
2.76	5.08	0.10	40.3	1.82	1.08
2.81	5.04	0.11	40.6	2.12	1.08

References 30

[1]	郑华. 柠檬酸的生产工艺及在食品中的应用[J]. 现代农业, 2009(5): 102-103.
	Zheng H. Production technology of citric acid and its application in food[J]. Modern Agriculture, 2009(5): 102-103.
[2]	梁恒宇, 林海龙, 武国庆, 等. 柠檬酸及其衍生物在洗涤剂工业中的应用[J]. 当代化工, 2015, 44(12): 2828-2831.
	Liang H Y, Lin H L, Wu G Q, et al. Application of citric acid and its derivatives in detergent industry[J]. Contemporary Chemical Industry, 2015, 44(12): 2828-2831.
[3]	贾菲, 郭一飞. 柠檬酸递送药物的研究进展[J]. 现代药物与临床, 2023, 38(8): 2080-2085.
	Jia F, Guo Y F. Advances in citric acid delivery of drugs[J]. Drugs & Clinic, 2023, 38(8): 2080-2085.
[4]	Salihu R, Abd Razak S I, Ahmad Zawawi N, et al. Citric acid: a green cross-linker of biomaterials for biomedical applications[J]. European Polymer Journal, 2021, 146: 110271.
[5]	彭超, 姚福伟, 朱威宇, 等. 柠檬酸发酵产业的市场分析与生产现状[J]. 当代化工, 2023, 52(9): 2196-2200.
	Peng C, Yao F W, Zhu W Y, et al. Analysis on production and market of critic acid fermentation industry[J]. Contemporary Chemical Industry, 2023, 52(9): 2196-2200.
[6]	Papagianni M. Advances in citric acid fermentation by Aspergillus niger: biochemical aspects, membrane transport and modeling[J]. Biotechnology Advances, 2007, 25(3): 244-263.
[7]	刘辰, 刘飞. 柠檬酸提取工艺的探索和氢钙法工业实践[J]. 精细与专用化学品, 2015, 23(1): 19-23.
	Liu C, Liu F. Development of citric acid extraction process and study on calcium hydrogen citrate precipitation method[J]. Fine and Specialty Chemicals, 2015, 23(1): 19-23.
[8]	Shakya A K, Pillai G, Chakrabarty S. Reinforcement learning algorithms: a brief survey[J]. Expert Systems with Applications, 2023, 231: 120495.
[9]	Buffet O, Pietquin O, Weng P. Reinforcement learning[M]//A Guided Tour of Artificial Intelligence Research. Cham: Springer International Publishing, 2020: 389-414.
[10]	Watkins C J C H, Dayan P. Q-learning[J]. Machine Learning, 1992, 8(3): 279-292.
[11]	Rummery G A, Niranjan M. On-line Q-learning Using Connectionist Systems[M]. Cambridge, UK: University of Cambridge, 1994.
[12]	Williams R J. Simple statistical gradient-following algorithms for connectionist reinforcement learning[J]. Machine Learning, 1992, 8(3): 229-256.
[13]	Timothy P L, Jonathan J H, Alexander P, et al. Continuous control with deep reinforcement learning[C]// International Conference on Learning Representations. SanDiego, 2015.
[14]	Gao X S, Yan L, Li Z J, et al. Improved deep deterministic policy gradient for dynamic obstacle avoidance of mobile robot[J]. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2023, 53(6): 3675-3682.
[15]	Ying F K, Liu H S, Jiang R X, et al. Trajectory generation for multiprocess robotic tasks based on nested dual-memory deep deterministic policy gradient[J]. IEEE/ASME Transactions on Mechatronics, 2022, 27(6): 4643-4653.
[16]	Du Y C, Chen J, Zhao C, et al. A hierarchical framework for improving ride comfort of autonomous vehicles via deep reinforcement learning with external knowledge[J]. Computer-Aided Civil and Infrastructure Engineering, 2023, 38(8): 1059-1078.
[17]	Ahmed M, Ahmed A, Chee P L, et al. Policy-based reinforcement learning for training autonomous driving agents in urban areas with affordance learning[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(8): 12562-12571.
[18]	Chen Y, Liu Z Y, Zhang Y C, et al. Deep reinforcement learning-based dynamic resource management for mobile edge computing in industrial internet of things[J]. IEEE Transactions on Industrial Informatics, 2021, 17(7): 4925-4934.
[19]	Zhang F Y, Yang Q Y, An D. CDDPG: a deep-reinforcement-learning-based approach for electric vehicle charging control[J]. IEEE Internet of Things Journal, 2021, 8(5): 3075-3087.
[20]	Matsushima T, Furuta H, Matsuo Y, et al. Deployment-efficient reinforcement learning via model-based offline optimization[EB/OL]. 2020, .
[21]	Yao Y, Xiao L, An Z C, et al. Sample efficient reinforcement learning via model-ensemble exploration and exploitation[C]//2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021: 4202-4208.
[22]	Brandi S, Fiorentini M, Capozzoli A. Comparison of online and offline deep reinforcement learning with model predictive control for thermal energy management[J]. Automation in Construction, 2022, 135: 104128.
[23]	Hochreiter S, Schmidhuber J. Long short-term memory[J]. Neural Computation, 1997, 9(8): 1735-1780.
[24]	Rumelhart D E, Hinton G E, Williams R J. Learning representations by back-propagating errors[J]. Nature, 1986, 323: 533-536.
[25]	Sherstinsky A. Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network[J]. Physica D: Nonlinear Phenomena, 2020, 404: 132306.
[26]	Rodriguez P, Cucurull G, Gonzalez J, et al. Deep pain: exploiting long short-term memory networks for facial expression classification[J]. IEEE Transactions on Cybernetics, 2022, 52(5): 3314-3324.
[27]	Liu Y W, Li D J, Wan S H, et al. A long short-term memory-based model for greenhouse climate prediction[J]. International Journal of Intelligent Systems, 2022, 37(1): 135-151.
[28]	Huang Y M, Dai X Y, Wang Q W, et al. A hybrid model for carbon price forecastingusing GARCH and long short-term memory network[J]. Applied Energy, 2021, 285: 116485.
[29]	Lindemann B, Müller T, Vietz H, et al. A survey on long short-term memory networks for time series prediction[J]. Procedia CIRP, 2021, 99: 650-655.
[30]	金其荣. 有机酸发酵工艺学[M]. 北京: 中国轻工业出版社, 1989.
	Jin Q R. Organic acid Fermentation Technology[M]. Beijing: China Light Industry Press, 1989.