Ruirui Zhong's Homepage

With the increasing attention to the state and role of people in intelligent manufacturing, there is a strong demand for human-cyber-physical systems (HCPS) that focus on human-robot interaction. The existing intelligent manufacturing system cannot satisfy efficient human-robot collaborative work. However, unlike machines equipped with sensors, human characteristic information is difficult to be perceived and digitized instantly. In view of the high complexity and uncertainty of the human body, this paper proposes a framework for building a human digital twin (HDT) model based on multimodal data and expounds on the key technologies. Data acquisition system is built to dynamically acquire and update the body state data and physiological data of the human body and realize the digital expression of multi-source heterogeneous human body information. A bidirectional long short-term memory and convolutional neural network (BiLSTM-CNN) based network is devised to fuse multimodal human data and extract the spatiotemporal features, and the human locomotion mode identification is taken as an application case. A series of optimization experiments are carried out to improve the performance of the proposed BiLSTM-CNN-based network model. The proposed model is compared with traditional locomotion mode identification models. The experimental results proved the superiority of the HDT framework for human locomotion mode identification.

Industry 5.0 is conceived to achieve the expected high level of smart manufacturing and the more proactive and adaptive human-robot collaboration (HRC). In particular, human-robot handover tasks (HRHTs) are a notably crucial aspect of HRC, and resolving how to motivate robots to comprehend human handover intentions constitutes an exigent issue that demands resolution. To address these problems, A framework that integrates hierarchical human digital twin (HHDT) and deep domain adaptation is proposed, where HHDT model is built to digitise the human physical and physiological state and enhance the robot’s perception of humans, a feature extractor-based on bidirectional long-short-term memory and spatio-temporal graph convolution network (BiLSTM-ST-GCN) is devised to extract latent spatio-temporal features of HRHTs intentions. Additionally, the framework incorporates a deep domain adaptation layer (DDAL) that facilitates knowledge transfer of HRHTs intentions by accounting for the distribution differences between the source and target operator objects. Extensive experiments demonstrate the superior performance of the proposed framework for HRHTs intention recognition and extraction of domain invariant features.