A deep learning approach for human gait recognition from time-frequency analysis images of inertial measurement unit signal
DOI:
https://doi.org/10.58190/ijamec.2023.44Keywords:
Biomechanical data, CNN, Human activity recognition, IMU, Time-frequency analysisAbstract
Biomechanical analysis using deep learning has been increasingly used in recent studies to identify human activity. Wearable sensor data from inertial measurement units (IMUs) is widely used for recognizing human activity, but has several drawbacks owing to its high volume and diversity. To overcome these issues, the time-domain and power spectral characteristics of IMU data can be extracted using digital signal processing (DSP) methods. Our research aimed to investigate time-frequency analysis (TFA) methods for classifying the spatio-temporal gait characteristics of physical walking performed by healthy subjects. In this study, open-source biomechanical sensor signal dataset was used. The DSP step was first carried out by segmenting IMU data from the four body segments of 22 healthy subjects, and then by applying Continuous Wavelet Transform (CWT) and Short Time Fourier Transform (STFT) methods. Moreover, the resultants of linear accelerometer signals were applied in a similar manner. The image datasets obtained from this step were applied to a deep convolutional neural network (CNN) model to classify human walking speed (WS) into three classes: fast, normal, and slow. The performance of the 2D-CNN model and the impact of DSP methods using IMU data were evaluated. In conclusion, the highest test classification results demonstrated that STFT-all (85.9%), CWT-all (79.3%), and CWT-resAcc (76.3%) based CNN models present a remarkably precise and easy-to-implement classification problem, with the highest test accuracy, when all IMU channels are subjected to STFT. The classification accuracies of 2D-CNN models were compared to commonly used ML models. The Deep CNN model is a useful gait evaluation tool for healthy subjects. Furthermore, it can enable the diagnosis and phase assessment of gait abnormalities and detect gait biomarkers in rehabilitative wearables.
Downloads
References
I. Hagoort, N. Vuillerme, T. Hortobágyi, and C. J. Lamoth, "Outcome-dependent effects of walking speed and age on quantitative and qualitative gait measures," Gait & Posture, vol. 93, pp. 39-46, 2022. DOI: https://doi.org/10.1016/j.gaitpost.2022.01.001.
T. Scone, M. Saadat, H. Barton, and A. Rastegarpanah, "Effects of Variations in Hemiparetic Gait Patterns on Improvements in Walking Speed," IRBM, p. 100733, 2022. DOI: https://doi.org/10.1016/j.irbm.2022.08.001.
M. De Vos, J. Prince, T. Buchanan, J. J. FitzGerald, and C. A. Antoniades, "Discriminating progressive supranuclear palsy from Parkinson's disease using wearable technology and machine learning," Gait & Posture, vol. 77, pp. 257-263, 2020. DOI: https://doi.org/10.1016/j.gaitpost.2020.02.007.
M. N. Favorskaya and V. V. Buryachenko, "Monocular depth-based visual tracker for gait recognition," Procedia Computer Science, vol. 207, pp. 205-214, 2022. DOI: https://doi.org/10.1016/j.procs.2022.09.053.
S. Gul, M. I. Malik, G. M. Khan, and F. Shafait, "Multi-view gait recognition system using spatio-temporal features and deep learning," Expert Systems with Applications, vol. 179, p. 115057, 2021. DOI: https://doi.org/10.1016/j.procs.2022.09.053.
B. Hu, S. Li, Y. Chen, R. Kavi, and S. Coppola, "Applying deep neural networks and inertial measurement unit in recognizing irregular walking differences in the real world," Applied Ergonomics, vol. 96, p. 103414, 2021. DOI: https://doi.org/10.1016/j.apergo.2021.103414.
C. F. Martindale, V. Christlein, P. Klumpp, and B. M. Eskofier, "Wearables-based multi-task gait and activity segmentation using recurrent neural networks," Neurocomputing, vol. 432, pp. 250-261, 2021. DOI: https://doi.org/10.1016/j.neucom.2020.08.079.
R. Soangra, "Multi-day longitudinal assessment of physical activity and sleep behavior among healthy young and older adults using wearable sensors," Irbm, vol. 41, no. 2, pp. 80-87, 2020. DOI: https://doi.org/10.1016/j.irbm.2019.10.002.
W. Teufl, M. Lorenz, M. Miezal, B. Taetz, M. Fröhlich, and G. Bleser, "Towards inertial sensor based mobile gait analysis: Event-detection and spatio-temporal parameters," Sensors, vol. 19, no. 1, p. 38, 2018. DOI: https://doi.org/10.3390/s19010038.
A. M. Nasrabadi, A. R. Eslaminia, P. R. Bakhshayesh, M. Ejtehadi, L. Alibiglou, and S. Behzadipour, "A new scheme for the development of IMU-based activity recognition systems for telerehabilitation," Medical Engineering & Physics, vol. 108, p. 103876, 2022. DOI: https://doi.org/10.1016/j.medengphy.2022.103876
T. Wang et al., "Accelerometer-based human fall detection using sparrow search algorithm and back propagation neural network," Measurement, vol. 204, p. 112104, 2022. DOI: https://doi.org/10.1016/j.measurement.2022.112104.
M. C. Ruder, M. A. Hunt, J. M. Charlton, T. Calvin, and D. Kobsar, "Validity and reliability of gait metrics derived from researcher-placed and self-placed wearable inertial sensors," Journal of Biomechanics, vol. 142, p. 111263, 2022. DOI: https://doi.org/10.1016/j.jbiomech.2022.111263
Q. T. Huynh and B. Q. Tran, "Time-Frequency Analysis of Daily Activities for Fall Detection," Signals, vol. 2, no. 1, pp. 1-12, 2021. DOI: https://doi.org/10.3390/signals2010001
Y. Jia, Y. Guo, G. Wang, R. Song, G. Cui, and X. Zhong, "Multi-frequency and multi-domain human activity recognition based on SFCW radar using deep learning," Neurocomputing, vol. 444, pp. 274-287, 2021. DOI: https://doi.org/10.1016/j.neucom.2020.07.136.
S. Tam, M. Boukadoum, A. Campeau-Lecours, and B. Gosselin, "A fully embedded adaptive real-time hand gesture classifier leveraging HD-sEMG and deep learning," IEEE transactions on biomedical circuits and systems, vol. 14, no. 2, pp. 232-243, 2019. DOI: https://doi.org/10.1109/tbcas.2019.2955641.
M. A. Ozdemir, D. H. Kisa, O. Guren, and A. Akan, "Hand gesture classification using time–frequency images and transfer learning based on CNN," Biomedical Signal Processing and Control, vol. 77, p. 103787, 2022. DOI: https://doi.org/10.1016/j.bspc.2022.103787
J. Too, A. R. Abdullah, and N. M. Saad, "Classification of hand movements based on discrete wavelet transform and enhanced feature extraction," International Journal of Advanced Computer Science and Applications, vol. 10, no. 6, 2019. DOI: https://doi.org/10.14569/ijacsa.2019.0100612.
N. Nahid, A. Rahman, and M. A. Ahad, "Deep learning based surface EMG hand gesture classification for low-cost myoelectric prosthetic hand," in 2020 Joint 9th International Conference on Informatics, Electronics & Vision (ICIEV) and 2020 4th International Conference on Imaging, Vision & Pattern Recognition (icIVPR), 2020: IEEE, pp. 1-8. DOI: https://doi.org/10.1109/icievicivpr48672.2020.9306613.
L. Chen, J. Fu, Y. Wu, H. Li, and B. Zheng, "Hand gesture recognition using compact CNN via surface electromyography signals," Sensors, vol. 20, no. 3, p. 672, 2020. DOI: https://doi.org/10.3390/s20030672.
N. U. Ahamed et al., "Using machine learning and wearable inertial sensor data for the classification of fractal gait patterns in women and men during load carriage," Procedia Computer Science, vol. 185, pp. 282-291, 2021. DOI: https://doi.org/10.1016/j.procs.2021.05.030.
C. Benegui and R. T. Ionescu, "Convolutional neural networks for user identification based on motion sensors represented as images," IEEE Access, vol. 8, pp. 61255-61266, 2020. DOI: https://doi.org/10.1109/access.2020.2984214.
C.-T. Yen, J.-X. Liao, and Y.-K. Huang, "Human daily activity recognition performed using wearable inertial sensors combined with deep learning algorithms," Ieee Access, vol. 8, pp. 174105-174114, 2020. DOI: https://doi.org/10.1109/access.2020.3025938.
D. Jung, M. D. Nguyen, M. Park, J. Kim, and K.-R. Mun, "Multiple classification of gait using time-frequency representations and deep convolutional neural networks," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 28, no. 4, pp. 997-1005, 2020. DOI: https://doi.org/10.1109/tnsre.2020.2977049
D. Jung et al., "Deep neural network-based gait classification using wearable inertial sensor data," in 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2019: IEEE, pp. 3624-3628. DOI: https://doi.org/10.1109/embc.2019.8857872
J. Camargo, A. Ramanathan, W. Flanagan, and A. Young, "A comprehensive, open-source dataset of lower limb biomechanics in multiple conditions of stairs, ramps, and level-ground ambulation and transitions," Journal of Biomechanics, vol. 119, p. 110320, 2021. DOI: https://doi.org/10.1016/j.jbiomech.2021.110320.
J. Huang, B. Chen, B. Yao, and W. He, "ECG arrhythmia classification using STFT-based spectrogram and convolutional neural network," IEEE access, vol. 7, pp. 92871-92880, 2019. DOI: https://doi.org/10.1109/access.2019.2928017
M. Kaushik, N. Baghel, R. Burget, C. M. Travieso, and M. K. Dutta, "SLINet: Dysphasia detection in children using deep neural network," Biomedical Signal Processing and Control, vol. 68, p. 102798, 2021.
J. Camargo, W. Flanagan, N. Csomay-Shanklin, B. Kanwar, and A. Young, "A machine learning strategy for locomotion classification and parameter estimation using fusion of wearable sensors," IEEE Transactions on Biomedical Engineering, vol. 68, no. 5, pp. 1569-1578, 2021. DOI: https://doi.org/10.1109/tbme.2021.3065809
I. Boukhennoufa, X. Zhai, V. Utti, J. Jackson, and K. D. McDonald-Maier, "Wearable sensors and machine learning in post-stroke rehabilitation assessment: A systematic review," Biomedical Signal Processing and Control, vol. 71, p. 103197, 2022. DOI: https://doi.org/10.1016/j.bspc.2021.103197
K. Bhakta, J. Camargo, W. Compton, K. Herrin, and A. Young, "Evaluation of continuous walking speed determination algorithms and embedded sensors for a powered knee & ankle prosthesis," IEEE Robotics and Automation Letters, vol. 6, no. 3, pp. 4820-4826, 2021. DOI: https://doi.org/10.1109/lra.2021.3068711
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 International Journal of Applied Methods in Electronics and Computers
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
