Informatica

Objective: The objective of this paper is to investigate which function is most appropriately describing the learning curve produced by C4.5 algorithm.

Methods: The J48 implementation of the C4.5 algorithm was applied to datasets (n=121) from publicly available repositories (e.g. UCI) in step wise k-fold cross-validation. First, four different functions (power, linear, logarithmic, exponential) were fit to the measured error rates. Where the fit was statistically significant (n=86), we measured the average mean squared error rate for each function and its rank. The dependent samples T-test was performed to test whether the differences between mean squared error are significantly different, and Wilcoxon's signed rank test was used to test whether the differences between ranks are significant.

Results: The decision trees error rate can be successfully modeled by an exponential function. In a total of 86 datasets, exponential function was a better descriptor of error rate function in 64 of 86 cases, power was best in 13, logarithmic in 3, and linear in 6 out of 86 cases. Average mean squared error across all datasets was 0.052954 for exponential function, and was significantly different at P=0.001 from power and at P=0.000 from linear function. The results also show that exponential function's rank is significantly different at any reasonable threshold (P=0.000) from the rank of any other model.

Conclusion: Our findings are consistent with tests performed in the area of human cognitive performance, e.g. with works by Heathcote et al. (2000), who were observing that the exponential function is best describing an individual learner. In our case we did observe an individual learner (C4.5 algorithm) at different tasks. The work can be used to forecast and model the future performance of C4.5 when not all data have been used or there is a need to obtain more data for better accuracy.

In the paper, we present an approach for an early assessment of the extracted knowledge (classification models) in the terms of performance (accuracy). The key questions are answered by detecting the point of convergence, i.e., where the classification model's performance does not improve any more even when adding more data items to the learning set. For the learning process termination criteria we developed a set of equations for detection of the convergence that follow the basic principles of the learning curve. The developed solution was evaluated on real datasets. The results of the experiment prove that the solution is well-designed: the learning process stopping criteria are not subjected to local variance and the convergence is detected where it actually has occurred.