Our method considers three factors to identify crowd behaviours: motion intensity, quantity and motion direction of pixels moving toward and away from certain regions. First we define basic motion structures of crowd aggregation and crowd dispersion. Based on integral optical flow, we then define four motion maps to describe pixel motions at each position, i.e. statistical analysis of quantity and motion direction of pixels moving toward or away from each position. After that, we introduce regional motion indicators to analyse motion at region-level, which is appropriate for crowd behaviour identification. At last, we use threshold segmentation to identify above-mentioned crowd behaviours. General scheme of our method is shown in Fig. 1.

Human behaviours reflect one’s emotion which can be affected by surroundings. Individual behaviours may not be reliable for indicating things happening around, but behaviours of crowd in public places are very reliable indicators. From human nature point of view, people react rapidly to urgent events, thus their motions become fast on video. For example, when people see something dangerous, they get away from the corresponding region in order to prevent getting hurt. Crowds not only react to their surroundings, they may also create emergencies. When conflict is going to happen between two groups of people, e.g. fans of two football clubs, they tend to approach each other rapidly. There are many other situations causing these crowd behaviours.

Crowd behaviours are related to regions spatially. As a region comprises several adjacent positions, before looking into those behaviours, one should describe motion at each position clearly. Basic optical flow and integral optical flow record basic information of motions, i.e. starting position and ending position of pixels, thus for each position in the frame, not only number of pixels passing through can be counted, their comprehensive moving directions can also be computed.

It makes more sense to analyse motions at region-level when studying crowd behaviours. In this section, meaningful indicators are defined and computed for regions of interest, these indicators will be used to identify crowd aggregation and crowd dispersion.

Methods proposed in this paper apply to identifying crowd behaviours from videos captured by stationary camera and with foreground mainly consisting of pedestrians. Videos captured by Closed Circuit Television (CCTV) in public places normally have these characteristics.

As described in Section 3.2, crowd aggregation and crowd dispersion are good indicators of abnormal or urgent events. When trying to identify these behaviours, lower limits of region size and regional motion intensity and thresholds for pixel quantity and motion direction should be properly determined according to specific purposes in different applications. One proper way to determine these parameters, which we adopt in our experiments, is to perform statistical analysis of videos which contain crowd aggregation or crowd dispersion, especially for the same scene.

We may have different types of regions of interest. Sometimes only several certain regions are of interest, other times the whole scene needs to be monitored. For the first case, indicators of Definitions 6–11 can be computed only for certain regions; for the second case, a sliding window should be moved through each position which is treated as the centre of a region. There are basically no restrictions on region shape, though normally self-symmetric regions should be used, e.g. squares, rectangles, circles, etc. Since the first case is just a special case of the second and much easier, we will just discuss the second case.

Square region will be used and its centre will represent the region. Let r denote a certain square region, c denote the centre position of r, then we have M I t ( c ) = M I t ( r ), IRQ t ( c ) = IRQ t ( r ), ORQ t ( c ) = ORQ t ( r ), IOI t ( c ) = IOI t ( r ), IS t ( c ) = IS t ( r ), OS t ( c ) = OS t ( r ). Thus M I t , IRQ t , ORQ t , IOI t , IS t , OS t can be seen as maps and are called regional motion intensity map (MI map), regional in-pixel relative quantity map (IRQ map), regional out-pixel relative quantity map (ORQ map), regional in/out indicator map (IOI map), regional in-pixel symmetry map (IS map) and regional out-pixel symmetry map (OS map), respectively. Regional versions of ICM map and OCM map, called regional in-pixel comprehensive motion map (RICM map) and regional out-pixel comprehensive motion map (ROCM map), respectively, can also be created to show how symmetrically pixels move toward and away from a region. Let RICM t and ROCM t denote RICM map and ROCM map, respectively, then RICM t ( c ) = RICM t ( r ) = 1 n ∑ p ∈ r ICM t ( p ), ROCM t ( c ) = ROCM t ( r ) = 1 n ∑ p ∈ r O C M t ( p ), where n is position number in r.

In order to describe crowd motion precisely, especially to determine whether they move symmetrically or not, the window size for computing basic optical flow should be about or a little smaller than the size of a person on video. Except for SCENE 7, there are two types of view for videos in Agoraset: top view and perspective view. In top view video, size of a person almost doesn’t change during motion. In perspective view video, size of a person becomes bigger when that person moves closer to the camera. Therefore, window size for computing optical flow is determined according to the size of a person away from the lens for perspective view video.

Munsell colour system is referenced to display integral optical flow. Different colours mean different motion directions (Fig. 5). For example, when a pixel moves right, then it will be red in the colour image of integral optical flow. In the meantime, colour purity depends on magnitude of displacement vector, the bigger the purer.

Images transformed from other intermediate results will be selectively shown based on the type of crowd behaviour to be identified. Transformation methods for these results are listed as follows: (1)

IQ map, OQ map, MI map, IRQ map, ORQ map, IOI map, IS map and OS map are transformed into grey images.

As shown in Fig. 6(a) and 6(c), the crowd starts to aggregate at the centre of the scene. The integral optical flow in Fig. 6(b) clearly shows their motion trend. Because interpolative motion path is used, the slender greyish-white trails in Fig. 6(e) mean values are very big at corresponding positions on IQ map. There are some noises at the lower right corner of IOI map causing most of Fig. 6(g) is black, but segmentation result in Fig. 6(h) shows there are more pixels moving toward than moving away from around the centre and some other places. Figure 6(i) shows pixels move symmetrically around the centre with light colours. Figure 6(j) shows their symmetrical motion directions much clearer. Final result is shown in Fig. 6(k) with green mask at the centre of regions in which crowd aggregation is happening. In order to get better visual result, the result is mixed with I t + 20 instead of  I t . When restriction on the symmetry of motion directions is relaxed, more regions are identified as shown in Fig. 6(l).

IS t which stands for symmetry of motion directions is a decisive factor in this example. This doesn’t mean motion intensity or pixel quantity can be ignored. For perspective view of SCENE 8 ( 512 × 512, 25 fps, angle of view: 30), same parameters are used and results are shown in Fig. 7.

In SCENE 7 ( 640 × 480, 30 fps, angle of view: 54.43), two group of pedestrians walk rapidly toward each other in two opposite directions. With the same parameters used above, results are shown in Fig. 8.

In top view of SCENE 5 ( 640 × 480, 30 fps), the crowd disperse from the lower right part of a square with walls. As the crowd disperse, motion intensity inside the corresponding region reduces, so later dispersion won’t be able to be detected unless increasing region size or reducing threshold for MI map. Interpolative motion path is used for motion analysis. i t v = 20 is frame interval parameter for computing integral optical flow, size of square region used to identify crowd behaviour is 101 × 101. t 21 = 10, t 22 = 2.0, t 23 = 0.8, t 24 = 4 are thresholds for MI map, ORQ map, IOI map, OS map, respectively.

As shown in Fig. 9(a) and 9(c), the crowd starts to disperse from the lower right part of the square. The integral optical flow in Fig. 9(b) clearly shows their motion trend. Because the dispersion happens near the bottom wall and pedestrians are stopped by the wall, thus symmetry of motion directions is affected, Fig. 9(j) shows the distortion. But combining with other segmentation results, this problem can be solved. Dispersion centre is clearly shown in Fig. 9(b), 9(e), 9(i). Final result is shown in Fig. 9(k) with blue mask at the centre of regions in which crowd dispersion is happening. Figure 9(l) shows result with symmetry restriction relaxed.

For perspective view of SCENE 5 ( 640 × 480, 30 fps, angle of view: 54.43), same parameters are used as above and results are shown in Fig. 10.

In SCENE 7, which is already used for crowd aggregation identification experiment, two groups of pedestrians continue their motion after aggregation, thus causing crowd dispersion later. Same parameters are used as above. As shown in Fig. 11(a) and 11(b), the crowd is blocked on the right side, therefore there is a much smaller area marked on the right side in identification result shown in Fig. 11(c). When restriction on symmetry of motion directions is relaxed, area around the centre where more pedestrians move upside than those move downside is marked as shown in Fig. 11(d).

We also test our method on real world videos downloaded from YouTube. Figure 12(a) shows that a team of South Korean police move fast in one direction to help the front police in a riot control exercise. Figure 12(b) shows the police and the protesters approach each other with the police occupying the position first. Figure 12(c) shows that the police who are confronting with the protesters start to extend their formation and the protesters start to retreat. The video shakes thus background motion is a little intensive, but with proper thresholds we still manage to get good results. Figure 12(d) shows that prisoners move fast in a Mexican prison riot, the blue area shows some of the prisoners start to disperse to dodge something thrown toward them (a white object in the red circle) from the dark area between the two buildings. Figure 12(e) shows many Marathon runners run along a U-shape street.

To the best of our knowledge, most of the non-training works published only detected abnormal behaviours or events, they didn’t identify specific crowd behaviours. Solmaz et al. (2012) proposed a method to identify five crowd behaviours including crowd dispersion and aggregation. Therefore our method is to be compared with this method.

It is important to notice that definitions in Solmaz et al. (2012) are not the same as ours, although some of those definitions are similar to ours to some extent. In the comparison, we take blocking and bottleneck in Solmaz et al. (2012) as crowd aggregation, and fountainhead as crowd dispersion. To estimate performance, we first generate ground truth manually, then count total number, correctly identified number, missed number and misidentified number of corresponding structure defined in Section 3.2 of each crowd behaviour. Table 1 shows the results.

As shown in Table 1, our method outperforms method in Solmaz et al. (2012). However, we have to point out that the definitions of crowd behaviour used here are in favour of our method. As a matter of fact, generally accepted definitions of crowd behaviour are not present yet, these can only be obtained through more researchers’ work in computer vision and related fields. We believe that, for abnormal or urgent event detection, it is important not to miss identification of related crowd behaviours. In our experiment, we choose thresholds in order to achieve this goal. The relatively high misidentified number is an apparent side effect of our choice.

(1)

Although we can get good results for jittering videos through changing thresholds in some cases, but that is not universal, thus our method should be applied to stable videos generally. In different applications thresholds for motion intensity, quantity and motion direction of pixels should be set depending on the scene and specific purposes.