Abstract
The experimental analysis on the performance of a proposed method is a crucial and necessary task to carry out in a research. In this contribution we focus on the use of statistical inference for analyzing the results obtained in a design of experiment within the field of computational intelligence. We present some studies involving a set of techniques in different topics which can be used for doing a rigorous comparison among the algorithms studied in an experimental comparison.
Particularly, we study whether the sample of results from multiple trials obtained by the run of several algorithms checks the required conditions for being analyzed through parametric tests. In most of the cases, the results indicate that the fulfillment of these conditions are problem dependent and indefinite, which justifies the need of using nonparametric statistics in the experimental analysis. We show a case study which illustrates the use of nonparametric tests and finally we give some guidelines on the use of nonparametric statistical tests.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Aguilar-Ruiz, J.S., Giráldez, R., Riquelme, J.C.: Natural encoding for evolutionary supervised learning. IEEE Transactions on Evolutionary Computation 11(4), 466–479 (2007)
Bacardit, J.: Pittsburgh genetic-based machine learning in the data mining era: representations, generalization and run-time. Phd Thesis, Dept. Computer Science University Ramon Llull, Barcelona, Spain (2004)
Ben-David, A.: A lot of randomness is hiding in accuracy. Engineering Applications of Artificial Intelligence 20, 875–885 (2007)
Corcoran, A.L., Sen, S.: Using real-valued genetic algorithm to evolve rule sets for classification. In: Proceedings of the IEEE Conference on Evolutionary Computation, pp. 120–124 (1994)
Eiben, A.E., Smith, J.E.: Introduction to Evolutionary Computing. Springer, Heidelberg (2003)
Goldberg, D.E.: Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley, Reading (1989)
Demšar, J.: Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research 7, 1–30 (2006)
Dietterich, T.G.: Approximate Statistical Test For Comparing Supervised Classification Learning Algorithms. Neural Computation 10(7), 1895–1923 (1998)
Fan, R.-E., Chen, P.-H., Lin, C.-J.: Working set selection using second order information for training SVM. Journal of Machine Learning Research 6, 1889–1918 (2005)
García, S., Molina, D., Lozano, M., Herrera, F.: A study on the use of nonparametric tests for analyzing the evolutionary algorithm’s behaviour: a case study on the CEC 2005 Special Session on Real Parameter Optimization. Journal of Heuristics (in press, 2008) DOI: 10.1007/s10732-008-9080-4
Molina, D., Herrera, F., Lozano, M.: Adaptive local search parameters for real-coded memetic algorithms. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC 2005), pp. 888–895 (2005)
Noether, G.E.: Sample size determination for some common nonparametric tests. Journal of American Statistical Association 82(398), 645–647 (1987)
Rojas, R., Feldman, J.: Neural Networks: A systematic introduction. Springer, Heidelberg (1996)
Shaffer, J.P.: Multiple hypothesis testing. Annual Review of Psychology 46, 561–584 (1995)
Sheskin, D.J.: Handbook of Parametric and Nonparametric Statistical Procedures. CRC Press, Boca Raton (2006)
Wilson, S.W.: Classifier fitness based on accuracy. Evolutionary Computation 3(2), 149–175 (1995)
Wolpert, D.H., Macready, W.G.: No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation 1(1), 67–82 (1997)
Wolpert, D.H.: The supervised learning no-free-lunch theorems. In: Proc. 6th Online World Conference on Soft Computing in Industrial Applications (2001)
Wright, S.P.: Adjusted p-values for simultaneous inference. Biometrics 48, 1005–1013 (1992)
Zar, J.H.: Biostatistical Analysis. Prentice-Hall, Englewood Cliffs (1999)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
García, S., Herrera, F. (2008). Design of Experiments in Computational Intelligence: On the Use of Statistical Inference. In: Corchado, E., Abraham, A., Pedrycz, W. (eds) Hybrid Artificial Intelligence Systems. HAIS 2008. Lecture Notes in Computer Science(), vol 5271. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-87656-4_3
Download citation
DOI: https://doi.org/10.1007/978-3-540-87656-4_3
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-87655-7
Online ISBN: 978-3-540-87656-4
eBook Packages: Computer ScienceComputer Science (R0)Springer Nature Proceedings Computer Science