## 1 Introduction to Open Set Problem

*et al.*, 2013), where there are usually only small amount of labelled data with a large amount of unlabelled data. Semi-supervised learning falls between unsupervised learning and supervised learning, and it can learn from both labelled and unlabelled instances. This can be combined by active method, like active clustering based classification method, which clusters both the labelled and unlabelled data with the guidance of labelled instances, then queries the label of the most informative instances in an active learning phase and after that classifies the data set (Szűcs and Henk, 2015).

*et al.*, 2014), i.e. classification problems in which the recognition system has to be robust to unseen categories. Formally, given

*K*known classes (categories) in the training set, and the task is not only to classify the new instances into known categories, but also to recognize when an instance does not belong to any of the known classes. This new category is called unknown class, thus the test set contains $K+1$ classes. The task is an extended version of the single-label classification, because after training

*K*classes the decision should be drawn among $K+1$ alternatives.

## 2 Related Work

*et al.*, 2001; Cevikalp and Triggs, 2012), RO-SVM (Zhang and Metaxas, 2006) determines the instance labels, and the rejection region during the training phase simultaneously. Furthermore, in the literature binary classification models have been proposed specifically for open set visual recognition tasks. Scheirer

*et al.*(2014) developed a Compact Abating Probability model (CAP model), where the probability of class membership decreases in value (abates) as points move from known data toward open space. Based on the CAP model, they described a new variant of SVM, the novel Weibull-calibrated SVM (W-SVM) for open set recognition, which combines useful properties of statistical extreme value theory for score calibration with one-class and binary SVMs. Scheirer

*et al.*(2014) claim that W-SVM outperforms their previous solutions, namely the 1-vs-Set Machine (Scheirer

*et al.*, 2013) and the ${P_{I}}$-SVM (Jain

*et al.*, 2014); besides, they included several other approaches in their experimental evaluation, which were all outperformed by W-SVM. In this paper we compare our solution to the W-SVM and discuss the results (see Section 6.3). The 1-vs-Set Machine algorithm (Scheirer

*et al.*, 2013) sculpts the decision space from the marginal distance of one-class or binary SVM with a linear kernel, so that it can reduce open space risk. This approach simply assigns class labels to instances during test. On the other hand, ${P_{I}}$-SVM (Jain

*et al.*, 2014) is developed for estimating the unnormalized posterior probability of class inclusion. The idea is based on knowledge of rejection the large set of unknown classes even under an assumption of incomplete class knowledge if an accurate model could be built for positive data for any known class without overfitting. The solution is formulated as modelling positive training data at the decision boundary, where the statistical extreme value theory can help. Bendale and Boult (2015) defined Open World recognition and presented the Nearest Non-Outlier (NNO) algorithm which adds object categories incrementally while detecting outliers and managing open space risk.

## 3 Double Probability Model

### 3.1 Theoretical Model

*i*) for a new instance, as outputs of the prediction of the original classifiers, and based on them the probability of the

*i*th class can be expressed as we describe in Eq. (4) and the expression for the probability of class ${C_{K+1}}$ can be seen in Eq. (5).

*j*th test instance is formalized in Eq. (7).

### 3.2 Double Smoothing

*N*data: ${\mathit{score}_{1}},{\mathit{score}_{2}},\dots ,{\mathit{score}_{N}}$. Between ${\mathit{score}_{i}}$ and ${\mathit{score}_{i+1}}$ the value of CDF changes from $\frac{i}{N}$ to $\frac{(i+1)}{(N+2)}$, so the difference of them can cause the smoothing error ($se$), described in Eq. (8).

*N*becomes infinite. Let us denote the number of elements in the original (i.e. before smoothing) CDF of the

*j*th class by ${N_{j}}$. The maximal error caused by double smoothing can be derived by Eq. (10).

## 4 Image Classification

*et al.*, 2007; Chatfield

*et al.*, 2011; Lazebnik

*et al.*, 2006) for the mathematical representation of the images and we used SVM (Support Vector Machine) (Boser

*et al.*, 1992; Cortes and Vapnik, 1995; Chatfield

*et al.*, 2011) for classifier. We should note that the DPM can be used with any classification process, as long as it provides probability values for each possible category.

*j*th Gaussian component respectively, furthermore $K=256$. We calculated the

*λ*parameter with ML (Maximum Likelihood) estimation by using the iterative EM (Expectation Maximization) algorithm (Dempster

*et al.*, 1977; Tomasi, 2004). We performed K-means clustering (MacQueen, 1967) over all the descriptors with 256 clusters to get the initial parameter model for the EM. The next step was to create a descriptor that specifies the distribution of the visual code words in an image, called high-level descriptor. To represent an image with high-level descriptor, the GMM based Fisher vector (see Eq. (12)) was calculated (Perronnin and Dance, 2007; Reynolds, 2009). These vectors were the final representations (image descriptor) of the images. where $\log p(X\mid \lambda )$ is the probability density function introduced in Eq. (11),

*X*denotes the SIFT descriptors of an image and

*λ*represents the parameter of GMM ($\lambda =\{{\omega _{j}}{\mu _{j}}{o_{j}}|j=1\dots K\}$).

*et al.*, 1992; Cortes and Vapnik, 1995) with RBF (Radial Basis Function) kernel. The one-against-all technique was applied to extend the SVM for multi-class classification. We used Platt’s (Platt, 2000) approach as probability estimator, which is included in LIBSVM (A Library for Support Vector Machines) (Chang and Lin, 2011; Huang

*et al.*, 2006). At this point we can decide whether to use the Double Probability Model for filtering out the test samples that possibly came from a previously unseen category, or keep the original predictions of the classifier (SVM). The CDF and reverse CDF (Eqs. (2) and (3)) can be calculated based on the class membership probabilities (in a validation set).

## 5 New Goodness Indicators for Classification in Open Set Problem

*I*is an indicator function and its value is 1 if the condition in Equation (14) is true, otherwise 0. The

*K*and

*U*are the sets of known and unknown instances (in the test set), ${C_{K}}$, ${C_{U}}$ are the sets of known and unknown classes, respectively (the unknown label is only one class, but ${C_{K}}$ typically contains more known classes). Furthermore, ${Y_{i}}$ and ${Y^{\prime }_{i}}$ denote the real and predicted class label of the

*i*th image.

*O*refers for open set problem). The decision for a test sample is drawn among $K+1$ alternatives; thus with the ${\mathit{Accuracy}_{O}}$ we evaluate those decisions among $K+1$ categories.

## 6 Experimental Results

### 6.1 Experimental Environment

*et al.*, 2004) collection which consists of 8677 images from 101 categories; and we created numerous data sets by randomly sampling the classes from the total data set. These subsets fit into six different types. The training set was formed from 70% of the images in the randomly selected known classes, and the test set contains the other 30% images from the known classes complemented by all of the unknown images. The reason behind isolating the unknown images is that the learning system is not allowed to use them, so all unknown images are basically unknown test samples. We randomly selected some of the known classes, and repeated this operation 20 times, so that we can take the average of the 20 results. We chose two different unknown sets, and we defined three different numbers of known classes to sample, therefore we had total of six types, as can be seen in Table 1. As we mentioned previously, we had 20 data sets of every type, so total of 120 data sets. In the rest of the paper we will consider only the types, instead of the individual data sets one by one; later on when we present the results of the data types, we mean the averaged results of the 20 individuals.

##### Table 1

Name | Number of known classes | Unknown set |

Airplanes5 | 5 | airplanes |

Airplanes10 | 10 | airplanes |

Airplanes20 | 20 | airplanes |

Faces5 | 5 | faces + faces easy |

Faces10 | 10 | faces + faces easy |

Faces20 | 20 | faces + faces easy |

### 6.2 Evaluation of Double Probability Model

##### Fig. 1

*y*-axis and the percentage of the unknown test samples is on the

*x*-axis.

*Airplanes*: 0.741,

*Faces*5 : 0.730,

*Airplanes*10 : 0.551,

*Faces*10 : 0.611,

*Airplanes*20 : 0.234,

*Faces*20 : 0.550. We can see that in the majority of tests our proposed model detected more than half of the unknown test samples, moreover, in case of Airplanes5 and Faces5 only a quarter of the unknown set remained undetected.

*x*-axis of the diagrams above, it represents the percentage of the unknown test samples. In addition to the averaged metric we also included the Q1 and the Q3 (first and third quartiles) statistical indicators to give a comprehensive view about the performance of our model. Figure 1 already showed that the averaged results are better when we use DPM, but by looking at and comparing the Q1, Q3 values of ${\mathit{Accuracy}_{E}}$ in Tables 2–4 we can notice that even every Q1 and Q3 is higher in case of using Double Probability Model; moreover, in some cases the Q1 with DPM outperforms the Q3 without DPM. Based on these results we conclude that our proposed model efficiently filters out the unknown test samples.

##### Table 2

% | Airplanes5 | Faces5 | ||||||||||

Without DPM | With DPM | Without DPM | With DPM | |||||||||

AVG | Q1 | Q3 | AVG | Q1 | Q3 | AVG | Q1 | Q3 | AVG | Q1 | Q3 | |

0 | 0.547 | 0.419 | 0.634 | 0.647 | 0.557 | 0.720 | 0.531 | 0.398 | 0.630 | 0.730 | 0.638 | 0.807 |

5 | 0.517 | 0.396 | 0.601 | 0.639 | 0.547 | 0.720 | 0.502 | 0.376 | 0.596 | 0.718 | 0.620 | 0.792 |

10 | 0.490 | 0.377 | 0.569 | 0.629 | 0.534 | 0.716 | 0.475 | 0.357 | 0.567 | 0.707 | 0.587 | 0.792 |

15 | 0.463 | 0.355 | 0.536 | 0.619 | 0.523 | 0.716 | 0.449 | 0.336 | 0.534 | 0.701 | 0.564 | 0.792 |

20 | 0.436 | 0.334 | 0.505 | 0.610 | 0.502 | 0.716 | 0.423 | 0.318 | 0.502 | 0.686 | 0.530 | 0.792 |

25 | 0.409 | 0.313 | 0.474 | 0.601 | 0.485 | 0.716 | 0.397 | 0.297 | 0.472 | 0.672 | 0.518 | 0.792 |

30 | 0.382 | 0.292 | 0.444 | 0.592 | 0.464 | 0.716 | 0.370 | 0.277 | 0.440 | 0.657 | 0.494 | 0.792 |

35 | 0.354 | 0.272 | 0.412 | 0.582 | 0.437 | 0.716 | 0.344 | 0.258 | 0.408 | 0.643 | 0.463 | 0.792 |

40 | 0.327 | 0.251 | 0.380 | 0.570 | 0.412 | 0.716 | 0.318 | 0.238 | 0.378 | 0.625 | 0.427 | 0.792 |

45 | 0.300 | 0.230 | 0.348 | 0.558 | 0.382 | 0.715 | 0.291 | 0.218 | 0.346 | 0.610 | 0.389 | 0.792 |

50 | 0.273 | 0.210 | 0.317 | 0.543 | 0.361 | 0.702 | 0.265 | 0.199 | 0.315 | 0.593 | 0.344 | 0.792 |

##### Table 3

% | Airplanes10 | Faces10 | ||||||||||

Without DPM | With DPM | Without DPM | With DPM | |||||||||

AVG | Q1 | Q3 | AVG | Q1 | Q3 | AVG | Q1 | Q3 | AVG | Q1 | Q3 | |

0 | 0.635 | 0.561 | 0.717 | 0.676 | 0.609 | 0.727 | 0.643 | 0.601 | 0.734 | 0.745 | 0.654 | 0.851 |

5 | 0.602 | 0.532 | 0.679 | 0.658 | 0.607 | 0.709 | 0.609 | 0.570 | 0.695 | 0.723 | 0.620 | 0.830 |

10 | 0.571 | 0.504 | 0.645 | 0.642 | 0.590 | 0.686 | 0.578 | 0.539 | 0.659 | 0.704 | 0.604 | 0.797 |

15 | 0.539 | 0.477 | 0.608 | 0.620 | 0.556 | 0.668 | 0.545 | 0.509 | 0.623 | 0.686 | 0.588 | 0.767 |

20 | 0.508 | 0.448 | 0.573 | 0.602 | 0.524 | 0.643 | 0.514 | 0.480 | 0.587 | 0.663 | 0.573 | 0.758 |

25 | 0.476 | 0.420 | 0.537 | 0.580 | 0.503 | 0.606 | 0.482 | 0.450 | 0.550 | 0.642 | 0.567 | 0.734 |

30 | 0.444 | 0.393 | 0.501 | 0.560 | 0.491 | 0.575 | 0.449 | 0.420 | 0.513 | 0.624 | 0.528 | 0.734 |

35 | 0.412 | 0.365 | 0.465 | 0.536 | 0.469 | 0.559 | 0.417 | 0.390 | 0.476 | 0.604 | 0.494 | 0.711 |

40 | 0.381 | 0.337 | 0.430 | 0.514 | 0.441 | 0.544 | 0.385 | 0.361 | 0.440 | 0.577 | 0.459 | 0.676 |

45 | 0.349 | 0.309 | 0.394 | 0.490 | 0.414 | 0.521 | 0.353 | 0.330 | 0.404 | 0.551 | 0.414 | 0.655 |

50 | 0.318 | 0.281 | 0.359 | 0.463 | 0.382 | 0.485 | 0.321 | 0.301 | 0.367 | 0.527 | 0.381 | 0.634 |

##### Table 4

% | Airplanes20 | Faces20 | ||||||||||

Without DPM | With DPM | Without DPM | With DPM | |||||||||

AVG | Q1 | Q3 | AVG | Q1 | Q3 | AVG | Q1 | Q3 | AVG | Q1 | Q3 | |

0 | 0.671 | 0.607 | 0.701 | 0.705 | 0.644 | 0.757 | 0.668 | 0.604 | 0.710 | 0.718 | 0.652 | 0.763 |

5 | 0.637 | 0.576 | 0.666 | 0.675 | 0.614 | 0.753 | 0.634 | 0.573 | 0.674 | 0.700 | 0.640 | 0.752 |

10 | 0.604 | 0.545 | 0.630 | 0.648 | 0.589 | 0.724 | 0.601 | 0.543 | 0.639 | 0.682 | 0.635 | 0.744 |

15 | 0.570 | 0.516 | 0.596 | 0.618 | 0.561 | 0.688 | 0.567 | 0.513 | 0.603 | 0.663 | 0.622 | 0.733 |

20 | 0.537 | 0.485 | 0.560 | 0.589 | 0.533 | 0.654 | 0.534 | 0.483 | 0.568 | 0.645 | 0.594 | 0.723 |

25 | 0.503 | 0.455 | 0.526 | 0.558 | 0.503 | 0.616 | 0.501 | 0.453 | 0.532 | 0.625 | 0.570 | 0.714 |

30 | 0.470 | 0.425 | 0.491 | 0.526 | 0.471 | 0.590 | 0.467 | 0.422 | 0.497 | 0.605 | 0.542 | 0.704 |

35 | 0.436 | 0.394 | 0.456 | 0.493 | 0.439 | 0.550 | 0.434 | 0.392 | 0.462 | 0.583 | 0.508 | 0.691 |

40 | 0.403 | 0.364 | 0.421 | 0.460 | 0.407 | 0.513 | 0.401 | 0.362 | 0.426 | 0.560 | 0.472 | 0.678 |

45 | 0.369 | 0.334 | 0.386 | 0.428 | 0.378 | 0.478 | 0.367 | 0.332 | 0.390 | 0.537 | 0.435 | 0.664 |

50 | 0.336 | 0.303 | 0.351 | 0.395 | 0.347 | 0.434 | 0.334 | 0.302 | 0.355 | 0.509 | 0.400 | 0.640 |

##### Table 5

% | ${\mathit{Accuracy}_{O}}$ | ${P_{\mathit{filter}}}$ | ||||||||||

A5 | F5 | A10 | F10 | A20 | F20 | A5 | F5 | A10 | F10 | A20 | F20 | |

0 | 0.547 | 0.531 | 0.635 | 0.643 | 0.671 | 0.668 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |

5 | 0.562 | 0.545 | 0.632 | 0.640 | 0.649 | 0.663 | 0.108 | 0.069 | 0.126 | 0.076 | 0.129 | 0.143 |

10 | 0.573 | 0.559 | 0.632 | 0.641 | 0.630 | 0.660 | 0.186 | 0.132 | 0.235 | 0.151 | 0.268 | 0.263 |

15 | 0.586 | 0.576 | 0.627 | 0.642 | 0.609 | 0.655 | 0.259 | 0.198 | 0.314 | 0.222 | 0.365 | 0.357 |

20 | 0.598 | 0.587 | 0.625 | 0.640 | 0.589 | 0.652 | 0.325 | 0.251 | 0.392 | 0.284 | 0.445 | 0.445 |

25 | 0.612 | 0.602 | 0.622 | 0.641 | 0.568 | 0.648 | 0.389 | 0.308 | 0.460 | 0.346 | 0.511 | 0.516 |

30 | 0.626 | 0.615 | 0.620 | 0.644 | 0.545 | 0.645 | 0.447 | 0.364 | 0.523 | 0.409 | 0.562 | 0.578 |

35 | 0.640 | 0.629 | 0.618 | 0.648 | 0.524 | 0.641 | 0.500 | 0.417 | 0.579 | 0.467 | 0.612 | 0.629 |

40 | 0.652 | 0.640 | 0.616 | 0.646 | 0.501 | 0.638 | 0.547 | 0.466 | 0.627 | 0.517 | 0.654 | 0.677 |

45 | 0.664 | 0.654 | 0.614 | 0.646 | 0.482 | 0.635 | 0.590 | 0.517 | 0.674 | 0.564 | 0.701 | 0.719 |

50 | 0.675 | 0.663 | 0.612 | 0.647 | 0.461 | 0.630 | 0.632 | 0.563 | 0.715 | 0.610 | 0.739 | 0.757 |

### 6.3 Comparison with the Weibull-Calibrated SVM (W-SVM)

*et al.*(2014). We tested the W-SVM on each data sets (total of 120) and evaluated the ${\mathit{Accuracy}_{O}}$ and ${P_{\mathit{filter}}}$ metrics, then compared them to the ones given by DPM. Figures 2 and 3 show the ${\mathit{Accuracy}_{O}}$ and the ${P_{\mathit{filter}}}$ metrics, respectively. As can be seen in the diagrams below, DPM has better performance in case of each data set type than W-SVM, and this implies that it would (most likely) outperform all the other techniques that were tested in Scheirer

*et al.*(2014). Table 6 gives a summary of the comparison by presenting the values of ${\mathit{Accuracy}_{O}}$ and ${P_{\mathit{filter}}}$ for each data set types given by DPM and W-SVM.

*θ*one-class SVMs trained on positive examples and

*θ*one-against-all binary SVMs, where

*θ*denotes the number of classes. It has two parameters: one of them is ${\delta _{\tau }}$ (fixed to 0.001 for all experiments in Scheirer

*et al.*, 2014), which is used to adjust the minimum threshold to consider data points in CAP model, and ${\delta _{R}}$ is the level of confidence needed in the estimation of W-SVM. It is important to note that W-SVM was introduced and validated on LETTER and MNIST data sets, where the recognition rate is higher than in image collections that contain photos of outdoor, natural scenes. Therefore, we suspected that a parameter optimization is necessary before going on and testing the W-SVM on each data sets. We used a separate 10-class data set for the optimization and found that ${\delta _{\tau }}=0.1$ and ${\delta _{R}}=0.1$ is an appropriate setting for such type of images (the default setting of W-SVM is ${\delta _{\tau }}=0.001$ and ${\delta _{R}}=0.1$). We decided not to modify the value of ${\delta _{R}}$, because by systematically increasing or decreasing this parameter, the ${\mathit{Accuracy}_{O}}$ and ${P_{\mathit{filter}}}$ were not converging to a global maximum. On the other hand, increasing ${\delta _{\tau }}$ resulted better detection rate on the unknown test samples up to a point (${\delta _{\tau }}=0.1$), where the number of false positive detections became high and it started to decrease both the ${\mathit{Accuracy}_{O}}$ and ${P_{\mathit{filter}}}$ metrics. In Fig. 2, we present the results of W-SVM, which were produced by the default and the optimized settings; thereby the difference between these options were demonstrated and therefore Fig. 3 and Table 6 show only the results given by the optimized W-SVM.

##### Fig. 2

*y*-axis and the percentage of the unknown test samples is on the

*x*-axis.

##### Fig. 3

*y*-axis and the percentage of the unknown test samples is on the

*x*-axis.

##### Table 6

% | ${\mathit{Accuracy}_{O}}$ | ${P_{\mathit{filter}}}$ | Method | ||||||||||

A5 | F5 | A10 | F10 | A20 | F20 | A5 | F5 | A10 | F10 | A20 | F20 | ||

0 | 0.547 |
0.531 |
0.635 |
0.643 |
0.671 |
0.668 |
0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | DPM |

0.495 | 0.420 | 0.563 | 0.486 | 0.513 | 0.505 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | W-SVM | |

5 | 0.562 |
0.545 |
0.632 |
0.640 |
0.649 |
0.663 |
0.108 |
0.069 | 0.126 |
0.076 | 0.129 |
0.143 |
DPM |

0.499 | 0.432 | 0.554 | 0.480 | 0.500 | 0.502 | 0.088 | 0.073 |
0.095 | 0.080 |
0.063 | 0.126 | W-SVM | |

10 | 0.573 |
0.559 |
0.632 |
0.641 |
0.630 |
0.660 |
0.186 |
0.132 | 0.235 |
0.151 |
0.268 |
0.263 |
DPM |

0.502 | 0.444 | 0.546 | 0.475 | 0.488 | 0.499 | 0.162 | 0.139 |
0.170 | 0.147 | 0.121 | 0.228 | W-SVM | |

15 | 0.586 |
0.576 |
0.627 |
0.642 |
0.609 |
0.655 |
0.259 |
0.198 | 0.314 |
0.222 |
0.365 |
0.357 |
DPM |

0.505 | 0.455 | 0.539 | 0.469 | 0.476 | 0.497 | 0.230 | 0.199 |
0.236 | 0.210 | 0.175 | 0.315 | W-SVM | |

20 | 0.598 |
0.587 |
0.625 |
0.640 |
0.589 |
0.652 |
0.325 |
0.251 | 0.392 |
0.284 |
0.445 |
0.445 |
DPM |

0.509 | 0.465 | 0.531 | 0.464 | 0.464 | 0.494 | 0.295 | 0.254 |
0.295 | 0.266 | 0.227 | 0.390 | W-SVM | |

25 | 0.612 |
0.602 |
0.622 |
0.641 |
0.568 |
0.648 |
0.389 |
0.308 | 0.460 |
0.346 |
0.511 |
0.516 |
DPM |

0.513 | 0.476 | 0.523 | 0.458 | 0.451 | 0.491 | 0.355 | 0.310 |
0.349 | 0.318 | 0.277 | 0.457 | W-SVM | |

30 | 0.626 |
0.615 |
0.620 |
0.644 |
0.545 |
0.645 |
0.447 |
0.364 |
0.523 |
0.409 |
0.562 |
0.578 |
DPM |

0.516 | 0.487 | 0.515 | 0.452 | 0.439 | 0.488 | 0.412 | 0.362 | 0.399 | 0.368 | 0.325 | 0.516 | W-SVM | |

35 | 0.640 |
0.629 |
0.618 |
0.648 |
0.524 |
0.641 |
0.500 |
0.417 |
0.579 |
0.467 |
0.612 |
0.629 |
DPM |

0.520 | 0.498 | 0.507 | 0.447 | 0.427 | 0.485 | 0.465 | 0.413 | 0.447 | 0.415 | 0.372 | 0.569 | W-SVM | |

40 | 0.652 |
0.640 |
0.616 |
0.646 |
0.501 |
0.638 |
0.547 |
0.466 |
0.627 |
0.517 |
0.654 |
0.677 |
DPM |

0.523 | 0.509 | 0.499 | 0.442 | 0.415 | 0.483 | 0.515 | 0.462 | 0.491 | 0.458 | 0.418 | 0.617 | W-SVM | |

45 | 0.664 |
0.654 |
0.614 |
0.646 |
0.482 |
0.635 |
0.590 |
0.517 |
0.674 |
0.564 |
0.701 |
0.719 |
DPM |

0.527 | 0.520 | 0.491 | 0.436 | 0.402 | 0.480 | 0.564 | 0.509 | 0.535 | 0.501 | 0.464 | 0.661 | W-SVM | |

50 | 0.675 |
0.663 |
0.612 |
0.647 |
0.461 |
0.630 |
0.632 |
0.563 |
0.715 |
0.610 |
0.739 |
0.757 |
DPM |

0.530 | 0.531 | 0.483 | 0.430 | 0.390 | 0.477 | 0.608 | 0.554 | 0.576 | 0.541 | 0.509 | 0.702 | W-SVM |