Research in Sports Medicine and Technology

An Adaptive Closed-loop Control Based Machine Learning For Rehabilitation Parkinson’s Patients

Authors

Saeed Mollahoseini paghale 1
Milad Fallahzade 2
Mohammadreza Amirseyfadini 3

1 PhD Candidate in Sport Biomechanics, University of Mazandaran, Mazandaran, Iran.

2 PhD Candidate in Sport Biomechanics, Faculty of Sport Sciences, Shahid Bahonar University of Kerman, Kerman, Iran.

3 Associate Professor, Faculty of Sport Sciences, Shahid Bahonar University of Kerman, Kerman, Iran.

Abstract
Background and Aims: Controlling hand tremors in neurological disorders like Parkinson's has gotten a lot of attention in recent decades. The number of theories about closed-loop deep brain stimulation is rapidly growing. The goal of this work is to offer a machine learning-based automated closed loop system for the rehabilitation of Parkinson's patients with hand tremor symptoms.

Materials and Methods: In the current study, vibration was simulated using a mathematical model that included a muscle model, basal ganglia, cortex, and supplementary motor area. To manage hand tremor, the non-integer PID proportional controller, as well as the intelligent Proximal Policy Optimization (PPO) algorithm as a subset of reinforcement learning, are employed to adapt the coefficients.

Results: One of the advantages of the proposed method, aside from reducing hand tremor and automatic learning to use at various levels of the disease, which has yielded acceptable results when compared to other control methods, is its practical implementation in the real world due to the simplicity of the controller. The automatic adjustment of artificial intelligence network coefficients in the presented strategy (PPO) makes it simple to create intelligent system.

Conclusion: The proposed intelligent system significantly reduces the side effects of continuous brain stimulation in the open-loop manner stimulation, in addition to optimizing output signals such as hand tremor compared to other controllers and being usable for all levels of the disease due to its adaptability.

Keywords

Parkinson’s Disease (PD)
Deep Brain Stimulation (DBS)
Reinforcement Learning (RL)
Proximal Policy Optimization (PPO)
Santillán, M., R. Hernández-Pérez, and R. Delgado-Lezama, A numeric study of the noise-induced tremor in a mathematical model of the stretch reflex. Journal of theoretical biology, 2003. 222(1): p. 99-115. doi.org/10.1016/S0022-5193(03)00016-X [DOI:10.1016/S0022-5193(03)00016-X]
Santillán, M., R. Hernández-Pérez, and R. Delgado-Lezama, A numeric study of the noise-induced tremor in a mathematical model of the stretch reflex. Journal of theoretical biology, 2003. 222(1): p. 99-115. doi.org/10.1016/S0022-5193(03)00016-X [DOI:10.1016/S0022-5193(03)00016-X]
Rouhollahi, K., et al., Designing a robust backstepping controller for rehabilitation in Parkinson's disease: a simulation study. IET systems biology, 2016. 10(4): p. 136-146. doi: 10.1049/iet-syb.2015.0068 [DOI:10.1049/iet-syb.2015.0068]
Rouhollahi, K., et al., Designing a robust backstepping controller for rehabilitation in Parkinson's disease: a simulation study. IET systems biology, 2016. 10(4): p. 136-146. doi: 10.1049/iet-syb.2015.0068 [DOI:10.1049/iet-syb.2015.0068]
de Paor, A.M. and M.M. Lowery, Analysis of the mechanism of action of deep brain stimulation using the concepts of dither injection and the equivalent nonlinearity. IEEE Transactions on Biomedical Engineering, 2009. 56(11): p. 2717-2720. doi: 10.1109/TBME.2009.2019962 [DOI:10.1109/TBME.2009.2019962]
de Paor, A.M. and M.M. Lowery, Analysis of the mechanism of action of deep brain stimulation using the concepts of dither injection and the equivalent nonlinearity. IEEE Transactions on Biomedical Engineering, 2009. 56(11): p. 2717-2720. doi: 10.1109/TBME.2009.2019962 [DOI:10.1109/TBME.2009.2019962]
Faraji, B., et al., Optimal Canceling of the Physiological Tremor for Rehabilitation in Parkinson's disease. Journal of Exercise Science and Medicine, 2019. 11(2): p. 113-124. doi: 10.32598/JESM.11.2.7 [DOI:10.32598/JESM.11.2.7]
Faraji, B., et al., Optimal Canceling of the Physiological Tremor for Rehabilitation in Parkinson's disease. Journal of Exercise Science and Medicine, 2019. 11(2): p. 113-124. doi: 10.32598/JESM.11.2.7 [DOI:10.32598/JESM.11.2.7]
Dobkin, R.D., et al., Telephone-based cognitive-behavioral therapy for depression in Parkinson disease. Journal of Geriatric Psychiatry and Neurology, 2011. 24(4): p. 206-214. doi.org/10.1177/0891988711422529 [DOI:10.1177/0891988711422529]
Dobkin, R.D., et al., Telephone-based cognitive-behavioral therapy for depression in Parkinson disease. Journal of Geriatric Psychiatry and Neurology, 2011. 24(4): p. 206-214. doi.org/10.1177/0891988711422529 [DOI:10.1177/0891988711422529]
Awantha, W., et al. A novel soft glove for hand tremor suppression: Evaluation of layer jamming actuator placement. in 2020 3rd IEEE International Conference on Soft Robotics (RoboSoft). 2020. IEEE. doi: 10.1109/RoboSoft48309.2020.9115994 [DOI:10.1109/RoboSoft48309.2020.9115994]
Awantha, W., et al. A novel soft glove for hand tremor suppression: Evaluation of layer jamming actuator placement. in 2020 3rd IEEE International Conference on Soft Robotics (RoboSoft). 2020. IEEE. doi: 10.1109/RoboSoft48309.2020.9115994 [DOI:10.1109/RoboSoft48309.2020.9115994]
Mertz, L., Taking on essential tremor: new tools and approaches offer patients increased treatment options. IEEE pulse, 2016. 7(3): p. 20-25. doi: 10.1109/MPUL.2016.2538481 [DOI:10.1109/MPUL.2016.2538481]
Mertz, L., Taking on essential tremor: new tools and approaches offer patients increased treatment options. IEEE pulse, 2016. 7(3): p. 20-25. doi: 10.1109/MPUL.2016.2538481 [DOI:10.1109/MPUL.2016.2538481]
Gheisarnejad, M., et al., A Close loop multi-area brain stimulation control for Parkinson's Patients Rehabilitation. IEEE Sensors Journal, 2019. 20(4): p. 2205-2213. doi: 10.1109/JSEN.2019.2949862 [DOI:10.1109/JSEN.2019.2949862]
Gheisarnejad, M., et al., A Close loop multi-area brain stimulation control for Parkinson's Patients Rehabilitation. IEEE Sensors Journal, 2019. 20(4): p. 2205-2213. doi: 10.1109/JSEN.2019.2949862 [DOI:10.1109/JSEN.2019.2949862]
Faraji, B., et al., An Adaptive ADRC Control for Parkinson's Patients Using Machine Learning. IEEE Sensors Journal, 2020. doi:10.1109/JSEN.2020.3048588 [DOI:10.1109/JSEN.2020.3048588]
Faraji, B., et al., An Adaptive ADRC Control for Parkinson's Patients Using Machine Learning. IEEE Sensors Journal, 2020. doi:10.1109/JSEN.2020.3048588 [DOI:10.1109/JSEN.2020.3048588]
Faraji, B., et al., Smart Sensor Control for Rehabilitation in Parkinson's Patients. IEEE Transactions on Emerging Topics in Computational Intelligence, 2021. doi: 10.1109/TETCI.2020.3045483 [DOI:10.1109/TETCI.2020.3045483]
Faraji, B., et al., Smart Sensor Control for Rehabilitation in Parkinson's Patients. IEEE Transactions on Emerging Topics in Computational Intelligence, 2021. doi: 10.1109/TETCI.2020.3045483 [DOI:10.1109/TETCI.2020.3045483]
Rouhollahi, K., et al., Design of robust adaptive controller and feedback error learning for rehabilitation in Parkinson's disease: a simulation study. IET systems biology, 2017. 11(1): p. 19-29. doi: 10.1049/iet-syb.2016.0014 [DOI:10.1049/iet-syb.2016.0014]
Rouhollahi, K., et al., Design of robust adaptive controller and feedback error learning for rehabilitation in Parkinson's disease: a simulation study. IET systems biology, 2017. 11(1): p. 19-29. doi: 10.1049/iet-syb.2016.0014 [DOI:10.1049/iet-syb.2016.0014]
Ferleger, B., et al., Fully implanted adaptive deep brain stimulation in freely moving essential tremor patients. Journal of Neural Engineering, 2020. 17(5): p. 056026. doi.org/10.1088/1741-2552/abb416 [DOI:10.1088/1741-2552/abb416]
Ferleger, B., et al., Fully implanted adaptive deep brain stimulation in freely moving essential tremor patients. Journal of Neural Engineering, 2020. 17(5): p. 056026. doi.org/10.1088/1741-2552/abb416 [DOI:10.1088/1741-2552/abb416]
Aghababa, M.P., Optimal design of fractional-order PID controller for five bar linkage robot using a new particle swarm optimization algorithm. Soft Computing, 2016. 20(10): p. 4055-4067. doi.org/10.1007/s00500-015-1741-2 [DOI:10.1007/s00500-015-1741-2]
Aghababa, M.P., Optimal design of fractional-order PID controller for five bar linkage robot using a new particle swarm optimization algorithm. Soft Computing, 2016. 20(10): p. 4055-4067. doi.org/10.1007/s00500-015-1741-2 [DOI:10.1007/s00500-015-1741-2]
Mohammadi, M., et al., Semisupervised deep reinforcement learning in support of IoT and smart city services. IEEE Internet of Things Journal, 2017. 5(2): p. 624-635. doi:10.1109/JIOT.2017.2712560 [DOI:10.1109/JIOT.2017.2712560]
Mohammadi, M., et al., Semisupervised deep reinforcement learning in support of IoT and smart city services. IEEE Internet of Things Journal, 2017. 5(2): p. 624-635. doi:10.1109/JIOT.2017.2712560 [DOI:10.1109/JIOT.2017.2712560]
Rose, L., M.C. Bazzocchi, and G. Nejat, A model-free deep reinforcement learning approach for control of exoskeleton gait patterns. Robotica, 2021: p. 1-26. doi.org/10.1017/S0263574721001600 [DOI:10.1017/S0263574721001600]
Rose, L., M.C. Bazzocchi, and G. Nejat, A model-free deep reinforcement learning approach for control of exoskeleton gait patterns. Robotica, 2021: p. 1-26. doi.org/10.1017/S0263574721001600 [DOI:10.1017/S0263574721001600]
Wang, X., et al., Deep reinforcement learning-based rehabilitation robot trajectory planning with optimized reward functions. Advances in Mechanical Engineering, 2021. 13(12): p. 16878140211067011. doi.org/10.1177/16878140211067011 [DOI:10.1177/16878140211067011]
Wang, X., et al., Deep reinforcement learning-based rehabilitation robot trajectory planning with optimized reward functions. Advances in Mechanical Engineering, 2021. 13(12): p. 16878140211067011. doi.org/10.1177/16878140211067011 [DOI:10.1177/16878140211067011]
Fischer, F., et al., Reinforcement learning control of a biomechanical model of the upper extremity. Scientific Reports, 2021. 11(1): p. 1-15. doi.org/10.1038/s41598-021-93760-1 [DOI:10.1038/s41598-021-93760-1]
Fischer, F., et al., Reinforcement learning control of a biomechanical model of the upper extremity. Scientific Reports, 2021. 11(1): p. 1-15. doi.org/10.1038/s41598-021-93760-1 [DOI:10.1038/s41598-021-93760-1]
Xu, J., et al., A robotic system with reinforcement learning for lower extremity hemiparesis rehabilitation. Industrial Robot: the international journal of robotics research and application, 2021. doi.org/10.1108/IR-10-2020-0230 [DOI:10.1108/IR-10-2020-0230]
Xu, J., et al., A robotic system with reinforcement learning for lower extremity hemiparesis rehabilitation. Industrial Robot: the international journal of robotics research and application, 2021. doi.org/10.1108/IR-10-2020-0230 [DOI:10.1108/IR-10-2020-0230]
Nahvi, A., F. Bahrami, and S. Hemmati, INVESTIGATING DIFFERENT TARGETS IN DEEP BRAIN STIMULATION ON PARKINSON'S DISEASE USING A MEAN-FIELD MODEL OF THE BASAL GANGLIA-THALAMOCORTICAL SYSTEM. Journal of Mechanics in Medicine and Biology, 2012. 12(02): p. 1240004. doi.org/10.1142/S0219519412400040 [DOI:10.1142/S0219519412400040]
Nahvi, A., F. Bahrami, and S. Hemmati, INVESTIGATING DIFFERENT TARGETS IN DEEP BRAIN STIMULATION ON PARKINSON'S DISEASE USING A MEAN-FIELD MODEL OF THE BASAL GANGLIA-THALAMOCORTICAL SYSTEM. Journal of Mechanics in Medicine and Biology, 2012. 12(02): p. 1240004. doi.org/10.1142/S0219519412400040 [DOI:10.1142/S0219519412400040]
Pooyan, M. and F. Ghoreishian, An Improved Modeling of Parkinson's Tremor and Investigation of Some Approaches to Remove this Symptom. International Journal of Engineering Transactions B: Applications, 2021. 34(5).. doi.org/10.5829/ije.2021.34.05b.20 [DOI:10.5829/ije.2021.34.05b.20]
Pooyan, M. and F. Ghoreishian, An Improved Modeling of Parkinson's Tremor and Investigation of Some Approaches to Remove this Symptom. International Journal of Engineering Transactions B: Applications, 2021. 34(5).. doi.org/10.5829/ije.2021.34.05b.20 [DOI:10.5829/ije.2021.34.05b.20]
MashhadiMalek, M., et al., Are rigidity and tremor two sides of the same coin in Parkinson's disease? Computers in Biology and Medicine, 2008. 38(11-12): p. 1133-1139. doi.org/10.1016/j.compbiomed.2008.08.007 [DOI:10.1016/j.compbiomed.2008.08.007]
MashhadiMalek, M., et al., Are rigidity and tremor two sides of the same coin in Parkinson's disease? Computers in Biology and Medicine, 2008. 38(11-12): p. 1133-1139. doi.org/10.1016/j.compbiomed.2008.08.007 [DOI:10.1016/j.compbiomed.2008.08.007]
Haeri, M., Y. Sarbaz, and S. Gharibzadeh, Modeling the Parkinson's tremor and its treatments. Journal of theoretical biology, 2005. 236(3): p. 311-322. doi.org/10.1016/j.jtbi.2005.03.014 [DOI:10.1016/j.jtbi.2005.03.014]
Haeri, M., Y. Sarbaz, and S. Gharibzadeh, Modeling the Parkinson's tremor and its treatments. Journal of theoretical biology, 2005. 236(3): p. 311-322. doi.org/10.1016/j.jtbi.2005.03.014 [DOI:10.1016/j.jtbi.2005.03.014]
Tang, Y., et al., Optimum design of fractional order PIλDμ controller for AVR system using chaotic ant swarm. Expert Systems with Applications, 2012. 39(8): p. 6887-6896. doi.org/10.1016/j.eswa.2012.01.007 [DOI:10.1016/j.eswa.2012.01.007]
Tang, Y., et al., Optimum design of fractional order PIλDμ controller for AVR system using chaotic ant swarm. Expert Systems with Applications, 2012. 39(8): p. 6887-6896. doi.org/10.1016/j.eswa.2012.01.007 [DOI:10.1016/j.eswa.2012.01.007]
Gheisarnejad, M. and M.H. Khooban, Design an optimal fuzzy fractional proportional integral derivative controller with derivative filter for load frequency control in power systems. Transactions of the Institute of Measurement and Control, 2019. 41(9): p. 2563-2581. doi.org/10.1177/0142331218804309 [DOI:10.1177/0142331218804309]
Gheisarnejad, M. and M.H. Khooban, Design an optimal fuzzy fractional proportional integral derivative controller with derivative filter for load frequency control in power systems. Transactions of the Institute of Measurement and Control, 2019. 41(9): p. 2563-2581. doi.org/10.1177/0142331218804309 [DOI:10.1177/0142331218804309]
Du, W. and S. Ding, A survey on multi-agent deep reinforcement learning: from the perspective of challenges and applications. Artificial Intelligence Review, 2021. 54(5): p. 3215-3238. doi.org/10.1007/s10462-020-09938-y [DOI:10.1007/s10462-020-09938-y]
Du, W. and S. Ding, A survey on multi-agent deep reinforcement learning: from the perspective of challenges and applications. Artificial Intelligence Review, 2021. 54(5): p. 3215-3238. doi.org/10.1007/s10462-020-09938-y [DOI:10.1007/s10462-020-09938-y]
Ceron, J.S.O. and P.S. Castro. Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research. in International Conference on Machine Learning. 2021. PMLR. doi.org/10.48550/arXiv.2011.14826
Ceron, J.S.O. and P.S. Castro. Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research. in International Conference on Machine Learning. 2021. PMLR. doi.org/10.48550/arXiv.2011.14826
Yoo, H., et al., Reinforcement learning based optimal control of batch processes using Monte-Carlo deep deterministic policy gradient with phase segmentation. Computers & Chemical Engineering, 2021. 144: p. 107133. doi: 10.1016/j.compchemeng.2020.107133 [DOI:10.1016/j.compchemeng.2020.107133]
Yoo, H., et al., Reinforcement learning based optimal control of batch processes using Monte-Carlo deep deterministic policy gradient with phase segmentation. Computers & Chemical Engineering, 2021. 144: p. 107133. doi: 10.1016/j.compchemeng.2020.107133 [DOI:10.1016/j.compchemeng.2020.107133]
Zimmer, M. and P. Weng, Exploiting the sign of the advantage function to learn deterministic policies in continuous domains. arXiv preprint arXiv:1906.04556, 2019. doi.org/10.48550/arXiv.1906.04556 [DOI:10.24963/ijcai.2019/625]
Zimmer, M. and P. Weng, Exploiting the sign of the advantage function to learn deterministic policies in continuous domains. arXiv preprint arXiv:1906.04556, 2019. doi.org/10.48550/arXiv.1906.04556 [DOI:10.24963/ijcai.2019/625]
Rehan, M. and K.-S. Hong, Modeling and automatic feedback control of tremor: adaptive estimation of deep brain stimulation. PLoS One, 2013. 8(4): p. e62888. doi.org/10.1371/journal.pone.0062888 [DOI:10.1371/journal.pone.0062888]
Rehan, M. and K.-S. Hong, Modeling and automatic feedback control of tremor: adaptive estimation of deep brain stimulation. PLoS One, 2013. 8(4): p. e62888. doi.org/10.1371/journal.pone.0062888 [DOI:10.1371/journal.pone.0062888]
Bellini, G., et al., Clinical impact of deep brain stimulation on the autonomic system in patients with Parkinson's disease. Movement Disorders Clinical Practice, 2020. 7(4): p. 373-382.. doi.org/10.1002/mdc3.12938 [DOI:10.1002/mdc3.12938]
Bellini, G., et al., Clinical impact of deep brain stimulation on the autonomic system in patients with Parkinson's disease. Movement Disorders Clinical Practice, 2020. 7(4): p. 373-382.. doi.org/10.1002/mdc3.12938 [DOI:10.1002/mdc3.12938]
Rouhollahi, K., et al., Rehabilitation of the Parkinson's tremor by using robust adaptive sliding mode controller: a simulation study. IET Systems Biology, 2019. 13(2): p. 92-99. doi: 10.1049/iet-syb.2018.5043 [DOI:10.1049/iet-syb.2018.5043]
Rouhollahi, K., et al., Rehabilitation of the Parkinson's tremor by using robust adaptive sliding mode controller: a simulation study. IET Systems Biology, 2019. 13(2): p. 92-99. doi: 10.1049/iet-syb.2018.5043 [DOI:10.1049/iet-syb.2018.5043]
Sharma, R., P. Gaur, and A. Mittal, Performance analysis of two-degree of freedom fractional order PID controllers for robotic manipulator with payload. ISA transactions, 2015. 58: p. 279-291. doi.org/10.1016/j.isatra.2015.03.013 [DOI:10.1016/j.isatra.2015.03.013]
Sharma, R., P. Gaur, and A. Mittal, Performance analysis of two-degree of freedom fractional order PID controllers for robotic manipulator with payload. ISA transactions, 2015. 58: p. 279-291. doi.org/10.1016/j.isatra.2015.03.013 [DOI:10.1016/j.isatra.2015.03.013]
Faraji, B., et al., Optimal Canceling of the Physiological Tremor for Rehabilitation in Parkinson's disease. Journal of Exercise Science and Medicine, 2019. 11(2): doi.org/10.32598/JESM.11.2.7 [DOI:10.32598/JESM.11.2.7]
Faraji, B., et al., Optimal Canceling of the Physiological Tremor for Rehabilitation in Parkinson's disease. Journal of Exercise Science and Medicine, 2019. 11(2): doi.org/10.32598/JESM.11.2.7 [DOI:10.32598/JESM.11.2.7]
Han, S.-Y. and T. Liang, Reinforcement-Learning-Based Vibration Control for a Vehicle Semi-Active Suspension System via the PPO Approach. Applied Sciences, 2022. 12(6): p. 3078. doi.org/10.3390/app12063078 [DOI:10.3390/app12063078]
Han, S.-Y. and T. Liang, Reinforcement-Learning-Based Vibration Control for a Vehicle Semi-Active Suspension System via the PPO Approach. Applied Sciences, 2022. 12(6): p. 3078. doi.org/10.3390/app12063078 [DOI:10.3390/app12063078]
Hajihosseini, M., et al., DC/DC power converter control-based deep machine learning techniques: Real-time implementation. IEEE Transactions on Power Electronics, 2020. 35(10): p. 9971-9977. doi: 10.1109/TPEL.2020.2977765 [DOI:10.1109/TPEL.2020.2977765]
Hajihosseini, M., et al., DC/DC power converter control-based deep machine learning techniques: Real-time implementation. IEEE Transactions on Power Electronics, 2020. 35(10): p. 9971-9977. doi: 10.1109/TPEL.2020.2977765 [DOI:10.1109/TPEL.2020.2977765]