Discussion and results in empirical research

¶ … calculated f-measures, all of the transformation methods improved the performance of most classification methods, though in many cases performance did not vary considerably between untransformed (i.e. original) and transformed data or between the varied transformations and classification methods that were applied to the various data sets. There are certain notable exceptions, in particular data sets for which specific classification systems seemed wholly unsuited regardless of any transformation method applied, and also a few cases in which a transformation method greatly improved the performance of a classification method. The NTP2 data set and the DARPA data set both resulted in lower overall performance across transformation and classification methods, yet there is a substantial spread in the performance of transformation and classification methods.

For the NTP2 data set, the MLP classification system was the strongest performer overall, and performance was significantly improved for several classification systems through use of the PPFSCADA transformation method -- in the case of the SVM classification method, performance was almost doubled following this transformation. The PPFSCADA transformation method generally had greatest positive impact on performance across data sets and transformation methods, though it actually greatly reduced performance when the 7NN classification method was applied to the NTP2 data set. Performance on the DAPRA set was somewhat higher on average than the NTP2 data set, though the MLP classification method showed especially poor performance with this set.

Discussion

Overall, when performance was already relatively high for a data set (and this tended to be relatively consistent across classification methods), the transformation methods did not show tremendous variance, though in certain cases of difficult data sets particular transformation methods did have significant impacts on the performance of specific classification methods. Performance was increased by the PPFSCADA transformation method to the greatest degree, and the MLP classification method was similarly the highest performer across a greater number of data sets than other classification methods, though not to as significant degree as PPFSCADA outperformed other transformation methods, on average. Given these results, it is necessary to question the benefits of conducting additional transformation processing on data sets that show high performance without any transformation performed; depending on the resource and time intensity of the additional transformation, the benefit achieved by such a transformation might not be worthwhile and in many cases might not be significant enough to affect the outcome of other measures and analyses. On the other hand, poor performance in pre-transformation analysis can be greatly improved through transformation in some cases, and multiple transformations might be necessary to achieve the best results.