Informatica

Du et al. (2015) addressed the problem of trajectory overlapping. They constructed colour motion pseudo-images of uniform size by using a skeleton sequence (x, y, z) mapped sequentially to the three channels of a colour image (R, G, B). The motion information is placed sequentially in non-overlapping slices. Later, Yang et al. (2018) mentioned that human skeletal joints are not isolated from each other. Therefore, they proposed a tree-shaped skeleton joint arrangement, which better preserves spatial information. Liu et al. (2019) proposed a novel spatio-temporal image representation of skeleton joints called “Skepxels.” The method utilizes basic building blocks called “Skepxel” to construct skeleton images of arbitrary dimensions. These images encode fine-grained spatiotemporal information about the human skeleton across different frames. Additionally, multiple joint permutations are employed to enhance action recognition using spatio-temporal visual attention on skeleton image sequences. Ke et al. (2017) proposed a skeletal representation. Four base points are selected, and the 3D information of their connection points is represented in three slices each. Twelve grayscale images are then generated and fed into a multi-task learning network to learn the internal relationships among the selected base points for action classification. Caetano et al. (2019) noted that CNNs implicitly extract temporal human motion information from skeleton images. The position information and motion direction of the skeleton joints are computed by setting the period to capture the long-term joint interaction information in the action. Meanwhile, it introduces the arrangement of tree skeleton joints (Yang et al., 2018) to construct a new skeleton motion representation.

These methods partly solved the spatio-temporal overlapping problem, but some significant limitations remain: 1) Constructing a colour image requires the feature values to be within the range of 1–255, which makes it difficult to distinguish subtle actions and may obviously reduce feature accuracy. 2) Simple scaling and cropping can deform the action representation to achieve a uniform size required by CNN input. For example, a running action may be misclassified as walking when forced to resize to a fixed dimension as a colour motion image. To solve these problems, our previous work (Yao et al., 2022) proposed to utilize more joint information, displacements, angles, velocities, etc., to construct a compact 3D Dense Joint Motion Matrix (DJMM) in the form of floating-point numbers. This forms a DJMM with dimensions: number of joints × number of features × frame size, ensuring the preservation of raw data. However, efficiently generating a uniform-size input remains an urgent challenge.

At present, key frame extraction methods for HAR are mainly evolved from traditional summary extraction techniques. Due to the rich information in images, hand-crafted features such as HOG, SSIM, SIFT, and optical flow have been developed. Researchers have conducted extensive work on key frame extraction in RGB video (Zhao et al., 2024), which can be categorized into three parts: shot-based, sampling, and clustering-based techniques. Inspired by research in RGB video and the need for more efficient HAR, researchers began to study key frame extraction from human action videos. Li et al. (2023) identify 3D skeleton action sequences based on a series of meaningful movements, including changes in body posture direction and geometric features and improve motion analysis efficiency. Phan et al. (2021) constructed a motion variation curve by calculating the pixel intensity change between two consecutive frames and selecting the k highest localized extremes as the corresponding key frames.

The inter-frame sampling strategy is a commonly used method to extract key frames. However, it will likely lose frames with rich information. Similarly, simply using the information difference between adjacent frames to select key frames relies only on local information and fails to capture long-term dependencies. Thus, more and more researchers are exploring and proposing new algorithms to identify the most suitable key frames. For example, Gan et al. (2023) utilized deep learning techniques to select key frames. They extracted the motion and appearance information of each frame through CNN, labelled each frame with a score and fitted a curve. Then, the local maxima and minima on the curve were identified and corresponded to the key frames in the video. Zhao et al. (2023) proposed an improved k-means algorithm based on joint contribution weighting, which makes the key frame extraction pay more attention to the joints with more significant relative motion. For the skeleton data, k frames with the maximum inter-frame euclidean distance and their corresponding previous frames are selected. They are used to construct the left and right boundaries to form k clusters for initialization. One frame is selected as the cluster centre, and the euclidean distance is used for iteration until the centre no longer changes. Zhang et al. (2023) proposed a video key frame extraction method based on multi-feature fusion of quaternion fourier transform. The proposed method first extracts dynamic and static features from colour video sequences. Their fused phase spectra are then obtained according to the quaternion fourier transform. Gaussian filtering is used to remove noise in the fused phase spectra. Then, the inverse fourier transform is performed to obtain the fused feature map. Finally, an adaptive key frame selection criterion is constructed. The final key frame set is obtained by feature map selection. Chen et al. (2024) transformed the key frame selection problem into a multi-objective optimization problem for binary coding. An evaluation model based on domain information and the number of key frames was proposed, which can adaptively adjust the number of key frames according to the compression rate while preserving the temporal order of actions.

The method of cluster-based key frame extraction is superior to basic difference-maximization approaches. The reason for its advantage is that it iterates clusters by using spatial features as indicators and extracts a consistent number of key frames between different action types. This explains why it is widely adopted in HAR systems. However, clustering methods overemphasize spatial feature variations. This can easily lead to incorrect grouping of frames that are distant in time but similar in space. This will disrupt the preservation of action phase transitions (e.g. take-off vs. landing in jumps). Researchers have attempted to introduce temporal modelling. However, these approaches either require adjusting the threshold for specific actions or cannot maintain a uniform number of key frames between different action types.

A drawback of unsupervised key frame based clustering algorithms is that they rely solely on the euclidean distance between skeleton joints when performing clustering. This method may group frames that are far apart in time based on pose similarity, thereby disrupting the inherent temporal continuity of actions. These methods often disrupt temporal order during clustering, which can lead to misjudgments in HAR. Many current algorithms that maintain temporal stability of actions face a critical limitation, as they either fail to achieve scale-consistent key frames or introduce random errors during scale normalization. These methods often rely on random frame deletion or insertion, which carries the risk of losing informative frames and adding low-saliency frames. To address these issues and establish network inputs with a unified scale without compromising input quality, we propose the KKF algorithm.

The proposed KKF algorithm aims to select the K key frames that can best represent the motion trend of the entire action. Inspired by the previous experiments (Zhang et al., 2023), we employed the concepts of inter-frame difference and cluster mean frame. As indicated in Fig. 2, we take 15 skeleton joints as an example, and the position data of these joints in each frame are represented as the following 45-dimensional feature vector. In all the following equations, the i represents the frame number of the human behavioural skeleton, the n represents the n-th skeletal joint point of each frame, and the position information of each frame ${F_{i}}$ can be expressed as:

Although inter-frame difference methods effectively capture motion magnitude, they exhibit limitations in detecting subtle kinematic patterns like hand clapping. This paper introduces a hybrid method that combines joint angle and joint position features. It improves keyframe extraction accuracy, especially in fine-grained action recognition tasks. As shown in Fig. 3, we select the angle of each joint and map it across the three planes using the calculation methods in formulas (16) and (17). The angle information of each frame can also be represented ${A_{i}}$ as a 45-dimensional vector:

When extracting key frames, we apply min-max normalization to standardize the scale of both angle and position features to a $[0,1]$ range. This process ensures compatibility between features that originally have divergent value ranges (e.g., joint angles in radians vs. displacement in meters), allowing their joint utilization in the unified key frame selection metric.

The normalized data of each frame are summed to form a 90-dimensional vector ${W_{i}}$ for calculating the key frame algorithm index.

The inter-frame difference $({V_{i,j}})$ and frame-to-CMF distance (${V_{i,{c_{j}}}}$) can be expressed as where the ${W_{i}^{n}}$ represents the n-th dimension of the normalized vector and ${\textit{CMF}_{{c_{j}}}^{n}}$ represents the n-th dimension of the ${\textit{CMF}_{{c_{j}}}}$.

This indicates that a larger V value corresponds to a greater comprehensive difference between frames. The core objective of our KKF algorithm is to cluster frames with high spatio-temporal similarity into one category, while assigning frames with significant spatio-temporal differences to distinct clusters. To ensure temporal stability, the KKF algorithm first divides all frames evenly into K initialization intervals based on temporal order to form K clusters ${C_{i}}=\{{c_{1}},{c_{2}},\dots ,{c_{k-1}},{c_{k}}\}$. The first and last frames of each initialization interval are selected to form two sets: The right boundary set of the initialization cluster $\{{r_{{c_{1}}}},{r_{{c_{2}}}},\dots ,{r_{{c_{k-1}}}},{r_{{c_{k}}}}\}$, and the left boundary set $\{{l_{{c_{1}}}},{l_{{c_{2}}}},\dots ,{l_{{c_{k-1}}}},{l_{{c_{k}}}}\}$. This method satisfies the requirement of significant spatial differences in cluster initialization to reduce the time consumption of iterations.

The concept of a cluster mean frame (CMF) is proposed and defined in equations (7) and (8): where the ${l_{{c_{i}}}}$ and ${r_{{c_{i}}}}$ are the left and right boundaries of cluster ${c_{i}}$, the ${N_{{c_{i}}}^{n}}$ is the n-dimensional vector of the $CMF$ of cluster ${c_{i}}$. The CMF is derived from each dimension and mean of all frames in cluster ${c_{i}}$, which is the vector that can best comprehensively represent the spatial information in the cluster frame. The CMF facilitates computation of the inter-cluster and frame-to-cluster dissimilarities so that the obtained frames can optimally characterize the cluster, ensuring that the holistic action semantics is obtained to the greatest extent. To operationalize this principle, we iteratively refine the attribution of the boundary frames of adjacent clusters by calculating the CMF of each cluster to achieve the balance of spatial information for each cluster and avoid cross-time compression. Finally, we iteratively optimize the cluster boundary frames and select the frame closest to the cluster CMF as the key information frame. In this way, we can extract the key frames that represent human behaviour. The pseudo-code of the critical moment mode selection algorithm is given in Algorithm 1 (i.e. KKF).

The proposed method optimizes the trade-off between accuracy and efficiency by dynamically weighting kinematic metrics. During human action analysis tasks, there exist some classic key frames predefined by human kinematics. These moments are manifested as: 1) peak kinematic discontinuity, and 2) maximum spatial deviation. These frames are rich in dynamic information and crucial for understanding the whole action sequence. For example, we can define the peak kinematic discontinuity of human movement as: the frame with the most extensive variation $S(x)$ from the previous frame, while the maximum spatial deviation as the frame with the most extensive variation $L(x)$ from the first frame. We introduce two functions: $S(x)$ and $L(x)$. Specifically, $S(x)$ denotes the short-term motion difference between consecutive frames, formulated as

In contrast, $L(x)$ denotes the long-term spatial deviation of frame x relative to the initial frame, defined as

Taking the action of waving the hand twice as an example, the change curves of both movement and maximum spatial deviation are visualized in Fig. 4 $S(x)$ and $L(x)$. When a peak is detected in the curve, we directly set that cluster’s CMF value to the value of the special frame.

Following our previous work (Yao et al., 2022), the DJMM has solved the problem of motion information loss when joint motion trajectories overlap. Inspired by this approach, we use the key frames obtained by the KKF algorithm to extract features such as displacement, angle, velocity, and angular velocity of the corresponding skeleton. The feature calculations are based on relative coordinates, which enhances viewpoint robustness. Finally, the features are arranged into a uniform-scale KDJMM, allowing us to better capture the subtle movement changes.

The construction of KDJMM is shown in Fig. 5. Each frame contains information on 15 3D skeletal joints. Among them, 19 features are extracted for each joint on position, displacement, velocity, angle, and angular velocity, demonstrated in Fig. 5(a). Then, the constructed K 2D matrices are stacked together in the order of the extracted K key frames to form a temporally stable and uniform KDJMM, as shown in Fig. 5(b). The details of the feature calculation procedure are given below:

Deeper CNNs often demonstrate stronger learning capabilities on the constructed data, but they also require a larger input scale. This is because, during continuous convolution and pooling, the matrix scale progressively decreases until convolution becomes unfeasible. Although solutions were provided (residual blocks), deeper architectures inevitably demand more computational resources. To overcome this computational limitation, we designed a lightweight shallow neural network, retaining only three convolutional layers. Considering the size of the KDJMM, we reduced the kernel size, strides, and pooling window. Additionally, to accelerate convergence, we added a batch normalization layer after each convolutional layer. The specific network structure is shown in Fig. 6. In our experiments, we replace the final maximum pooling and flattening layers in the network with Spatial Pyramid Pooling (SPP) layers for multi-scale purpose. To further improve the representation quality, we introduced a CBAM ahead of the SPP layer, allowing the network to emphasize salient features before performing multi-scale training. The structures of the SPP and CBAM modules are jointly illustrated in Fig. 7.

Human action characterization must account for inherent biological variability. Consider two individuals executing identical motions – one standing 1.8 meters tall, the other 1.6 meters. Their height disparity introduces measurement artifacts. Furthermore, inter-subject kinematic variations manifest in three domains: temporal (movement pacing discrepancies), spatial (viewpoint-dependent morphological distortions), and directional (trajectory orientation effects). These observations motivate our multi-modal augmentation protocol incorporating camera perspective adjustments, anthropometric scaling, and velocity profile warping.

Considering that our feature construction choices are based on relative coordinates, the KDJMM is unaffected by viewpoint changes. In this experiment, we follow our previous work (Yao et al., 2020). Body size was controlled by multiplying all coordinates by a body size coefficient (BS). The speed of movement is controlled by the velocity coefficient (VS). We control the difference between the body shape and the speed around 0.2, BS = $(0.8,0.85,0.9,0.95,1.0,1.05,1.10,1.15,1.2)$ and VS = $(0.8,0.9,1.0,1.1,1.2,1.3)$. By combining the BS and the VS, the set of data will be enlarged up to 54 times of the original one. The diversity of data will be effectively increased, helping the model to accelerate the training process and learn a more comprehensive representation of features, thus improving the accuracy of classification or detection.

The proper setting of VS and BS coefficients is crucial. Taking a 1.7-meter-tall individual as an example, when BS = 1.5 is applied, their height would be scaled to 2.55 meters, which clearly exceeds normal human body dimensions. Similarly, excessively large VS coefficients would distort the speed characteristics of actions like walking and running, making it difficult to distinguish these basic motion patterns. We recommend maintaining these ranges: 0.8 ⩽ BS ⩽ 1.2 and 0.8 ⩽ VS ⩽ 1.3.

The FL-3D dataset was collected in 2012 at the University of Florence, using a Kinect camera, and contains nine activities: “wave, drink from a bottle, answer phone, clap, tie lace, sit down, stand up, read watch, bow.” Ten subjects performed each action two to three times, yielding 215 samples in total, with approximately 20–30 samples per action. For each subject, 15 skeletal joint points were recorded.

The UTK-3D dataset consists of depth sequences collected at 15 fps using a Kinect camera from the Windows SDK Beta version, providing RGB, depth, and 3D skeleton data. The dataset covers ten daily activities: “walking, sitting down, standing up, picking up, carrying, throwing, pushing, pulling, waving and clapping.” Ten subjects performed each action twice, but only 19 samples of the carrying action were available as the skeleton information for one sample was not captured. Twenty joints were recorded for each subject.

The HY-3D dataset was independently collected, using the Kinect v2.0 camera, and covered 15 simple movement types: “talking on the phone, drinking water, waving, looking at the clock, patting dust off clothes, falling, pushing a chair, jumping in place, standing still, clapping, walking, sitting down, sitting still, and sitting and clapping.” Nine subjects were asked to perform each of these 15 activities between three and four times. They recorded the 3D coordinates of all 25 joints provided by the Kinect v2.0 sensor, obtaining 413 samples, with each movement type represented by 35 to 37 samples. In addition, four complex behaviour types were collected to study the timing of the movements, namely “Sit – Stand up – Pat dust on clothes, Jump – Sit – Jump – Wave, Walk – Sit – Jump – Wave – Sit and Sit down and clap – Stand up and clap – Jump – Wave.” Fifteen of these simple movement types were used for our current study.

In this section, we evaluate the extraction of key frames by visualizing the clustering process, comparing the colour distributions of frames in the clusters produced by different clustering methods, and finally obtaining a sparse motion history map (SMHI). As demonstrated in Fig. 8, SMHIs effectively represent motion trajectories for elementary actions, validating the feasibility of algorithmically extracted key frames for HAR.

HAR differs from simple image classification. HAR not only relies on spatial information but also on temporal information. As shown in Fig. 8(b), traditional clustering methods (DBSCAN, HC, SC, K-means) cannot distinguish frames with the same spatial position but opposite velocity directions for periodic actions such as jumping in place in keyframe extraction. In the figure, frames with significant colour differences are clustered together. Figure 8 shows that the proposed method maintains temporal coherence. The cluster colours range from light to dark, and the method decomposes the “jumping in place” action into crouch preparation, take-off, and landing. This result verifies that the proposed method effectively addresses temporal disorder and accurately extracts the key frames of the action.

Considering that MHI is widely known in the field of HAR, and that the original MHI and the key frame extracted SMHI behave as the exact size of images, their computational efficiency as input to the model is the same. We used both representations to compare HAR accuracy, thereby evaluating the effectiveness of the key frames. Three scales of SMHI (SMHI-5, SMHI-10, SMHI-15) are extracted using angle and displacement. The accuracy of different clustering methods on the dataset and the time of extracting key frames are compared based on key frames to verify the validity of the algorithm. Then, the validity of KDJMM is demonstrated by comparing the performances of SMHI and KDJMM on the dataset. Throughout the experiment, 80% of the dataset is used for training and 20% for testing. The number of training rounds is set to 200. Meanwhile, an early stopping mechanism is employed to halt training when the loss falls below 0.005, preventing overfitting.

We evaluated our proposed method on three datasets. The experimental results of confusion matrices (Fig. 11) and overall accuracy (Table 8) demonstrated its effectiveness. Specifically, the confusion matrices for FL-3D and UTK-3D (Fig. 11a–b) show that our method not only classifies different action categories such as “clap, bow, and sit down”, but also correctly classifies most of the similar action categories, such as “stand up, sit down, and stand still”. Only the actions of “drink and carry” exhibited slightly lower recognition rates. However, the recognition accuracy of sit-still and check watch behaviours remains limited in the HY-3D dataset. The main reason is that there are many similar actions in HY-3D, such as “drink, check watch, call phone” or some composite actions contain some other atomic actions, such as “sit, sit still, sit-applaud”. These may bring ambiguity to our proposed method. To further investigate the cause of these confusions, we visualized some similar actions in the HY-3D dataset (Fig. 12). The actions like “drinking water, looking at a watch, making a phone call” are all share a highly similar arm-raising gesture towards the head. Similarly, all the actions of “sitting, sitting still, sitting and clapping” contains the action of sitting. Although the HY-3D dataset provides 25 joints per frame, our current model uses only 15 major body joints. Therefore, incorporating these additional joints into the model’s representation could provide distinctive features to mitigate these recognition errors.

Finally, comparative experiments with benchmark methods are conducted to objectively assess our method’s performance. The results in Table 8 show that our method achieves superior accuracy across all three datasets. At the same time, our method relies only on key frames for HAR, and it reduces processing time significantly.