This feature captures the spatial changes of a joint over a specific duration. Equation (1), which determines whether a frame is key, can also serve as a feature to represent actions. Notably, D represents the joint displacement in the 3D coordinate system. To obtain a larger-scale motion matrix, D is projected onto the x, y, and z axes in this study.

The physical or kinematic meaning of this feature is recording the direction variation of a joint in a certain frame’s duration. It contains both spatial and temporal information. The context information is essential to describe an action.

According to the vector’s translation in-variance, the motion direction in ( j − i ) frames’ duration can be projected on plan x y, x z and z y, and their values can be calculated by the law of cosines. (6) α i , j = α x y i , j = x Δ x 2 + Δ y 2 , α x z i , j = x z Δ x 2 + Δ z 2 , α z y i , j = z Δ z 2 + Δ y 2 , where { x , y , z } is the position of a certain joint in the ith frame, while { Δ x , Δ y , Δ z } represents the joint’s displacement along the x, y, and z-axes between the ith and jth frames.

This feature indicates the motion intensity of a certain joint in a period. From the physics perspective, it can be used to discriminate similar actions. Based on the data D, the velocity V can be readily computed using equation (7). (7) V = D 0.03333 × n , where 0.03333 is the duration time of one frame (the Kinect v2 record video at 30 fps, which indicates there are 30 frames in a second), and n is the number of interval frames.

In the geometrical view, the set of joint angles of a certain frame can be partly considered as a description of human pose, while an MHI-like global feature can be quantified by storing the angles in a float-point array by time sequence.

As it is shown in Figure 3, there are two types of joint angles. One is constituted by 3 adjacent joints and their related bones, and another one is constituted by the designed joints and red dotted lines (such as Head, HandTipLeft, HandTipRight, FootLeft and FootRight).

In many works, the joints’ coordinates, even the raw coordinates, are the only adopted data to describe an action. Our previous works (Yao et al., 2020) have proved 3 rules for action quantification and representation.

(1) The relative coordinates are more suitable than raw coordinates to represent human actions. Raw coordinates are highly variable and may be influenced by the camera’s installation position, leading to unintended consequences. Consequently, methods relying on raw coordinates are inherently dataset-oriented, often resulting in poor performance when the dataset varies or is applied in real-world scenarios.

(2) The necessary pre-processing of the coordinates is helpful to improve the accuracy. Despite CNNs’ exceptional feature extraction capabilities, the calculation of essential motion features, including speed, orientation, and joint angles, among others, can furnish CNNs with additional valuable data, ultimately leading to enhanced detection accuracy.

(3) It is necessary to define several temporal scales for action samples. Because the durations of different actions can vary significantly (for example, a ‘punch’ action may be completed in 10∼15 frames, while a ‘sit down’ action may take 30 ∼60 frames), only one uniformed scale may result in motion information loss or motion variation.

In this paper, we followed the above three rules and addressed a small-scale issue to further improve the flexibility and accuracy of our method. In most research endeavours, the action representation typically consists of an image or a float matrix. Given that existing algorithms or depth cameras can provide at most 25 joints, the resulting motion image or motion matrix is relatively small in size. With continuous advances in computing hardware, lightweight representation models have hit inherent performance bottlenecks. Their low-dimensional design, once a key advantage, now restricts accuracy improvements.

To meet all the requirements of the above analysis, a multi-original-point idea was raised to generate the motion matrices with flexible scales. As shown in Figure 4(a), displacement, direction, velocity, and angle are calculated on a frame-by-frame basis.

According to literature (Le et al., 2018), the relative origin point should be selected at the joint that exhibits minimal movement during action. Therefore, the 8 joints, such as Neck, SpineShoulder, ShoulderLeft, ShoulderRight, SpineBase, SpineCenter, HipLeft, HipRight, are adopted as the candidates of the relative origin points. As it is shown in Figure 4.

Using multiple relative original points has two advantages. First, one motion feature can be calculated multiple times. Thus, the scale of feature dimension can also be enlarged. Second, one motion feature based on different relative original points can partly reflect the geometrical information of the human body in a certain frame, which is also useful for the CNN to learn deep features.

In most of the “skeleton + CNN”-based methods, the dimensions of the motion representation are ‘joints × features × frames’. For example, with four motion features, a 45-frame action yields a representation of dimensions 25 × 4 × 45. If only 10 key frames are extracted, the dimensionality shrinks to 25 × 4 × 10, which is too small to support deep convolutional operations. However, if we take all the relative origin points to calculate the 4 motion features and use their variations, the scale can be enlarged. For example, the joint displacement of a joint in 3D coordinate can be reflected in three planes. Thus, one displacement feature can be divided into 4 features. Furthermore, if all the 8 relative origin points are adopted, the final scale of the action representation matrix can be enlarged to ( 25 × ( 4 × 4 ) × ( 10 × 8 ) ), which can be re-sized to ( 200 × 160 × 1 ).

In our method, an action is represented by a local feature form—MSMM, and a global feature form—SMHI. As Figure 1(d) shows, the MSMM is used as the input of the first-stage neural network because of its small size and the innate merits of local features. If the prediction confidence of the first-stage network exceeds a certain threshold, the prediction result is deemed final. However, if the predicting confidence is lower than the threshold value, the second-stage neural network starts working. The SMHI is fed into the second-stage network, which produces a supplementary prediction from a global semantic perspective. The final confidence score is computed as the average of the two stages’ outputs, and the class with the highest final confidence is selected as the recognition result.

The advantages of this network structure can be summarized in two aspects. First, the network uses both local feature and global feature. Local feature utilization in the first-stage network mitigates inherent limitations of global feature. In the second-stage network, global feature-based predictions offer semantic enhancements, boosting accuracy when first-stage confidence is lower than a specific threshold. Second, by adjusting the threshold of prediction confidence in the first-stage network, a clear distinction can be made between effectiveness and efficiency.

Just as its original version—MHI, SMHI is also a kind of global feature. Due to the fixed size of motion image, the input parameters that are based on MHI and SMHI will be the same. Therefore, there is no difference in the efficiency between MHI-based and SMHI-based methods. Hence, this section focuses solely on evaluating the effectiveness of SMHI-based methods and comparing it with MHI-based results. In the experiments, 80% of the samples are used for training, while 20% of them are used for testing. However, for SMHI, 80% of the samples are used for extracting key frames and generating SMHI, while in testing stage, all the samples are used for generating MHI and testing. Notably, to ensure experimental consistency and rigour, a stratified 80/20 train-test split is performed at the category level for each dataset, rather than a global random split applied to all samples collectively. The sample IDs for each partition are recorded and fixed. All experiments are conducted using the identical set of training samples for model training and the identical set of test samples for evaluation. Table 1 records the detailed results based on DenseNet121.

Table 1 demonstrates that all four motion image types can supply the neural network with essential motion data for HAR tasks. Furthermore, SMHI-D + A exhibited the best overall performance in experiments, confirming the efficacy of our key frame extraction algorithm. Experimental results indicate that 10 key frames of SMHI are optimal. Hence, in subsequent experiments, we will exclude 5 and 15 key frames of SMHI-D + A, focusing on SMHI-D + A in the following sections that stands for SMHI-D + A generated by 10 key frames.

Compared with global features, local features offer numerous advantages, especially when occlusion occurs. In our previous work (Yao et al., 2021), motion features were computed frame by frame to construct a 3D floating-point matrix called the Dense Joint Motion Matrix (DJMM), which served as the input to a self-defined CNN. The small size and flexibility of the motion matrix endow our previous works with great superiority in time efficiency and competitive achievements in accuracy. However, the bottleneck shows when further improvement is conducting. While the compact size of the motion matrix enhances efficiency, it poses a significant challenge in employing deeper neural networks to boost accuracy. In the proposed method, this bottleneck is broken by utilizing MSMM. Based on Figure 4(b), the subsequent experiments calculate MSMMs using the method of Mean Squared, which involves 8 relative origin points. Additionally, to facilitate multi-scale learning, the softmax layer in conventional CNNs is substituted by the SPP layer.

In the MSMM group of Table 2, it can be indicated that the test accuracies are generally superior to those in the DJMI group, though the networks also suffered from overfitting problems. In the experiments, there are 2 networks, VGG19 and Densenet121, which should be paid special attention to. Within the DJMI group, VGG19 in the DJMIM subset achieved a training accuracy of only 26.41%, suggesting that DJMM is not suitable for VGG19 or other deep networks. However, the MSMM group achieved a peak value of 100% in the training stage, demonstrating that a larger-scale MSMM is a superior choice for deep neural networks.

Table 2 also reveals that test accuracies are generally lower than those achieved during the training phase. This indicates that the neural networks exhibit overfitting. This pattern is particularly pronounced in early sequential architectures such as VGG16 and VGG19, as deeper network structures are more prone to overfitting. Furthermore, it is worth noting that data augmentation alone is insufficient to fully mitigate the overfitting problem. For example, as Table 3 shows, in the VGG19 model with DJMM input (tensor dimensions: 23 joints × 7 features × 9 key frames), the tensor size is reduced to ( … , 1 , … ) at the third convolutional block—a dimension too small to sustain subsequent convolutional operations.

To further prove this conclusion, the training processes are visualized and displayed in Figure 6. It is shown that the MSMM can help neural networks reach training purposes more quickly. As depicted in Figure 6(a), the network attained a training convergence at approximately 65 epochs, whereas in Figure 6(b), the network reached a training convergence at around 50 epochs. Furthermore, an analysis of the training curves of VGG16 and VGG19 reveals that adopting large-scale motion matrices is an effective strategy to alleviate prevalent deep learning obstacles and prevent neural networks from suffering overfitting induced by small-scale tensors. Additionally, the arbitrarily configurable count of relative origin points enables flexible tuning of the MSMM dimension. This characteristic is compatible with both shallow and deep neural network architectures and facilitates a favourable trade-off between computational efficiency and model performance.

The above experiments explained and proved 3 key points. 1) Is it feasible to use key frames to represent human actions? 2) How many key frames are the best choice to represent an action? 3) Is the proposed data structure—MSMM a better representation for describing human actions? However, the overfitting problem caused by insufficient training samples remains unresolved. Furthermore, the proposed method’s final HAR model has yet to undergo evaluation. In this section, we present the detailed solutions for addressing these two issues.

In our experiments, we developed a data augmentation program to simulate actions performed by subjects of different heights. The height scale factor was adjusted within [ 0.85 , 1.15 ] with a step size of 0.01. As a result, the number of training samples was increased by 30 times. Thus, there are 6 450 samples of Florence-3D, 5 970 samples of UT-3D, and 115 500 samples of HanYue-3D respectively. Assume that an adult’s height is about 1.7 meters, then the generated person’s height is among [ 1.45 , 1.96 ], which matches the height of people in the real world. Furthermore, based on the methodology of our proposed data augmentation technique, additional strategies can readily be employed to create more action samples, including altering motion directions and adjusting motion speeds. The threshold of the first stage is defined as 0.8.

Figure 7 records the training and testing results of DenseNet121 that evaluated on Florence-3D, UT-3D and HanYue-3D respectively. From the results, three points can be concluded. First, the smoothness of the curves indicates that MSMM is a good data structure for action recognition. Second, the high accuracies achieved in both training and testing stages have proven that an action can be adequately represented by several key frames, with other frames considered as redundant input that can be discarded. Third, either multi-scale learning or data augmentation has positive influence on HAR accuracy improvement (Xin et al., 2024). Moreover, since the nature of the proposed method is CNN, other prior technologies, including attention mechanisms, batch normalization and etc., can further contribute to accuracy improvement.

In the previous experiments, the proposed method has demonstrated its capacity and adaptability. However, the comprehensive evaluation of our network’s architecture has not been conducted.

In this section, the two-stage cascade structure of our neural network and data augmentation strategy were all adopted to show the best performance and superiority by comparing it with other typical methods. To further verify the outstanding performance of the proposed framework, Table 4 compares its performance with classical CNN-based methods, key frame-based methods, and our previous works across multiple evaluation dimensions. For all comparative approaches listed in the table, we strictly follow the hyper-parameter configurations reported in their original publications. Their evaluation metrics are directly extracted from published literature if available; for methods lacking official reported results, we replicate the experiments under a unified dataset split to obtain corresponding performance values.

For additional performance improvement of the proposed framework, we conducted exhaustive analysis on all mispredicted samples. Among these samples, several “wave hand” actions were incorrectly identified as the “answer phone” category.

For an intuitive visualization of such misclassification cases, Figure 8 selects 3 representative key frames of the samples and the corresponding JTMs of the two action categories. Since the Florence-3D dataset lacks annotations for left and right hand joints, the resultant JTMs extracted from these two distinct movements exhibit extremely similar feature distributions. Although this limitation can be effectively alleviated by introducing separate left- and right-hand joint coordinates to reconstruct new discriminative joint temporal maps (JTMs), neither the DJMM nor the MSMM framework achieves reliable recognition performance on publicly available datasets. Specifically, samples misclassified by the DJMM model cannot be correctly distinguished by the MSMM either.

Furthermore, regarding the efficiency improvement based on the DJMM method, a detailed analysis and proof have been conducted in our previous work (Yao et al., 2020), (Yao et al., 2021). The concept of MSMM is based on DJMM. Therefore, we will not repeat the proof of its efficiency improvement here.