1. Home
  2. Issues
  3. Volume 36, Issue 3 (2025)
  4. Performance Comparison of Single-Objecti ...

Informatica


Performance Comparison of Single-Objective Evolutionary Algorithms Implemented in Different Frameworks
Volume 36, Issue 3 (2025), pp. 677–712
Pub. online: 16 September 2025      Type: Research Article      Open accessOpen Access
Received
1 December 2024
Accepted
1 September 2025
Published
16 September 2025

Abstract

Fair comparison with state-of-the-art evolutionary algorithms is crucial, but is obstructed by differences in problems, parameters, and stopping criteria across studies. Metaheuristic frameworks can help, but often lack clarity on algorithm versions, improvements, or deviations. Some also restrict parameter configuration. We analysed source codes and identified inconsistencies between implementations. Performance comparisons across frameworks, even with identical settings, revealed significant differences, sometimes even with the authors’ own code. This questions the validity of comparisons using such frameworks. We provide guidelines to improve open-source metaheuristics, aiming to support more credible and reliable comparative studies.

References

 
Alba, E., Ferretti, E., Molina, J.M. (2007). The influence of data implementation in the performance of evolutionary algorithms. In: International Conference on Computer Aided Systems Theory. Springer, pp. 764–771.
 
Bartz-Beielstein, T., Doerr, C., van den Berg, D., Bossek, J., Chandrasekaran, S., Eftimov, T., Fischbach, A., Kerschke, P., La Cava, W., Lopez-Ibanez, M., Malan, K.M., Moore, J.H., Naujoks, B., Orzechowski, P., Volz, V., Wagner, M., Weise, T. (2020). Benchmarking in optimization: best practice and open issues. arXiv preprint arXiv:2007.03488.
 
Biedrzycki, R. (2021). Comparison with state-of-the-art: traps and pitfalls. In: 2021 IEEE Congress on Evolutionary Computation (CEC). IEEE, pp. 863–870.
 
Biscani, F., Izzo, D. (2020). A parallel global multiobjective framework for optimization: pagmo. Journal of Open Source Software, 5(53), 2338. https://doi.org/10.21105/joss.02338.
 
Blank, J., Deb, K. (2020). Pymoo: multi-objective optimization in python. Ieee Access, 8, 89497–89509.
 
Camacho Villalón, C.L., Stützle, T., Dorigo, M. (2020). Grey wolf, firefly and bat algorithms: three widespread algorithms that do not contain any novelty. In: International Conference on Swarm Intelligence. Springer, pp. 121–133.
 
Clerc, M., Kennedy, J. (2002). The particle swarm-explosion, stability, and convergence in a multidimensional complex space. IEEE Transactions on Evolutionary Computation, 6(1), 58–73.
 
Črepinšek, M., Liu, S.-H., Mernik, M. (2014). Replication and comparison of computational experiments in applied evolutionary computing: common pitfalls and guidelines to avoid them. Applied Soft Computing, 19, 161–170.
 
Darwish, A. (2018). Bio-inspired computing: algorithms review, deep analysis, and the scope of applications. Future Computing and Informatics Journal, 3(2), 231–246.
 
Deb, K., Deb, D. (2014). Analysing mutation schemes for real-parameter genetic algorithms. International Journal of Artificial Intelligence and Soft Computing, 4(1), 1–28.
 
Deb, K., Agrawal, R.B., (1995). Simulated binary crossover for continuous search space. Complex Systems, 9(2), 115–148.
 
Ding, K., Tan, Y. (2014). Comparison of random number generators in particle swarm optimization algorithm. In: 2014 IEEE Congress on Evolutionary Computation (CEC). IEEE, pp. 2664–2671.
 
Dokeroglu, T., Sevinc, E., Kucukyilmaz, T., Cosar, A. (2019). A survey on new generation metaheuristic algorithms. Computers & Industrial Engineering, 137, 106040.
 
Durillo, J.J., Nebro, A.J. (2011). jMetal: a Java framework for multi-objective optimization. Advances in Engineering Software, 42(10), 760–771.
 
Dzalbs, I. (2021). OptPlatform: metaheuristic optimisation framework for solving complex real-world problems. PhD thesis, Brunel University London.
 
EARS (2019). EARS – Evolutionary Algorithms Rating System (Github), available at https://github.com/UM-LPM/EARS.
 
Eftimov, T., Korošec, P. (2019). Identifying practical significance through statistical comparison of meta-heuristic stochastic optimization algorithms. Applied Soft Computing, 85, 105862.
 
Eiben, A.E., Jelasity, M. (2002). A critical note on experimental research methodology in EC. In: Proceedings of the 2002 Congress on Evolutionary Computation. CEC’02 (Cat. No. 02TH8600), Vol. 1. IEEE, pp. 582–587.
 
Ezugwu, A.E., Adeleke, O.J., Akinyelu, A.A., Viriri, S. (2020). A conceptual comparison of several metaheuristic algorithms on continuous optimisation problems. Neural Computing and Applications, 32, 6207–6251. https://doi.org/10.1007/s00521-019-04132-w.
 
Faris, H., Aljarah, I., Mirjalili, S., Castillo, P.A., Guervós, J.J.M. (2016). EvoloPy: an open-source nature-inspired optimization framework in python. IJCCI (ECTA), 1, 171–177.
 
Fortin, F.-A., De Rainville, F.-M., Gardner, M.-A., Parizeau, M., Gagné, C. (2012). DEAP: evolutionary algorithms made easy. Journal of Machine Learning Research, 13, 2171–2175.
 
Funakoshi, T., Nojima, Y., Ishibuchi, H. (2016). Effects of different implementations of a real random number generator on the search behavior of multiobjective evolutionary algorithms. In: 2016 Joint 8th International Conference on Soft Computing and Intelligent Systems (SCIS) and 17th International Symposium on Advanced Intelligent Systems (ISIS). IEEE, pp. 172–177.
 
Glickman, M.E. (1999). Parameter estimation in large dynamic paired comparison experiments. Journal of the Royal Statistical Society Series C: Applied Statistics, 48(3), 377–394.
 
Glickman, M.E. (2012). Example of the Glicko-2 system. Boston University.
 
Goldberg, D. (1991). What every computer scientist should know about floating-point arithmetic. ACM Computing surveys (CSUR), 23(1), 5–48.
 
Hadka, D. (2014). MOEA Framework: A Free and Open Source Java Framework for Multiobjective Optimization. Available at http://moeaframework.org.
 
Hansen, N., Ostermeier, A. (2001). Completely derandomized self-adaptation in evolution strategies. Evolutionary computation, 9(2), 159–195.
 
Hansen, N., Akimoto, Y., Baudis, P. (2019). CMA-ES/pycma on Github. Zenodo. https://doi.org/10.5281/zenodo.2559634.
 
Hansen, N., Auger, A., Finck, S., Ros, R. (2010). Real-Parameter Black-Box Optimization Benchmarking 2010: Experimental Setup. Research Report RR-7215, INRIA. https://hal.inria.fr/inria-00462481.
 
Hansen, N., Auger, A., Ros, R., Mersmann, O., Tušar, T., Brockhoff, D. (2021). COCO: a platform for comparing continuous optimizers in a black-box setting. Optimization Methods and Software, 36(1), 114–144.
 
Holland, J.H. (1992). Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence. MIT press.
 
Kadavy, T., Viktorin, A., Kazikova, A., Pluhacek, M., Senkerik, R. (2022). Impact of boundary control methods on bound-constrained optimization benchmarking. IEEE Transactions on Evolutionary Computation, 26(6), 1271–1280.
 
Kalami Heris, S. (2019). Yarpiz Evolutionary Algorithms Toolbox (YPEA).
 
Karaboga, D., Basturk, B. (2007). A powerful and efficient algorithm for numerical function optimization: artificial bee colony (ABC) algorithm. Journal of global optimization, 39(3), 459–471.
 
Kennedy, J., Eberhart, R. (1995). Particle swarm optimization. In: Proceedings of ICNN’95 – International Conference on Neural Networks, Vol. 4, pp. 1942–1948.
 
Kovačević, Ž., Ravber, M., Liu, S.-H., Črepinšek, M. (2022). Automatic compiler/interpreter generation from programs for domain-specific languages: code bloat problem and performance improvement. Journal of Computer Languages, 70, 101105.
 
LaTorre, A., Molina, D., Osaba, E., Del Ser, J., Herrera, F. (2020). Fairness in bio-inspired optimization research: a prescription of methodological guidelines for comparing meta-heuristics. arXiv preprint arXiv:2004.09969.
 
Lee, H.M., Jung, D., Sadollah, A., Kim, J.H. (2020). Performance comparison of metaheuristic algorithms using a modified Gaussian fitness landscape generator. Soft Computing, 24, 7383–7393. https://doi.org/10.1007/s00500-019-04363-y. https://link.springer.com/article/10.1007/s00500-019-04363-y.
 
Ma, Z., Vandenbosch, G.A. (2012). Impact of random number generators on the performance of particle swarm optimization in antenna design. In: 2012 6th European conference on antennas and propagation (EUCAP). IEEE, pp. 925–929.
 
Ma, Z., Wu, G., Suganthan, P.N., Song, A., Luo, Q. (2023). Performance assessment and exhaustive listing of 500+ nature-inspired metaheuristic algorithms. Swarm and Evolutionary Computation, 77, 101248.
 
Merelo, J., Romero, G., Arenas, M.G., Castillo, P.A., Mora, A.M., Laredo, J.L.J. (2011). Implementation matters: programming best practices for evolutionary algorithms. In: International Work-Conference on Artificial Neural Networks. Springer, pp. 333–340.
 
Merelo-Guervós, J.-J., Blancas-Álvarez, I., Castillo, P.A., Romero, G., Rivas, V.M., García-Valdez, M., Hernández-Águila, A., Romáin, M. (2016a). A comparison of implementations of basic evolutionary algorithm operations in different languages. In: 2016 IEEE Congress on Evolutionary Computation (CEC). IEEE, pp. 1602–1609.
 
Merelo-Guervós, J.J., Blancas-Alvarez, I., Castillo, P.A., Romero, G., García-Sánchez, P., Rivas, V.M., García-Valdez, M., Hernández-Águila, A., Román, M. (2016b). Ranking the performance of compiled and interpreted languages in genetic algorithms. In: Proceedings of the International Conference on Evolutionary Computation Theory and Applications, Porto, Portugal, Vol. 11, pp. 164–170.
 
Mernik, M., Liu, S.-H., Karaboga, D., Črepinšek, M. (2015). On clarifying misconceptions when comparing variants of the artificial bee colony algorithm by offering a new implementation. Information Sciences, 291, 115–127.
 
Mirjalili, S., Mirjalili, S.M., Lewis, A. (2014). Grey wolf optimizer. Advances in Engineering Software, 69, 46–61.
 
Molina, D., Poyatos, J., Del Ser, J., García, S., Hussain, A., Herrera, F. (2020). Comprehensive taxonomies of nature-and bio-inspired optimization: inspiration versus algorithmic behavior, critical analysis recommendations. Cognitive Computation, 12(5), 897–939.
 
Morales-Castañeda, B., Pérez-Cisneros, M., Cuevas, E., Zaldívar, D., Toski, M., Rodríguez, A. (2025). Analyzing metaheuristic algorithms performance and the causes of the zero-bias problem: a different perspective in benchmarks. Evolutionary Intelligence, 18(2), 1–19.
 
Niu, P., Niu, S., liu, N., Chang, L., (2019). The defect of the Grey Wolf optimization algorithm and its verification method. Knowledge-Based Systems, 171, 37–43.
 
Osaba, E., Villar-Rodriguez, E., Del Ser, J., Nebro, A.J., Molina, D., LaTorre, A., Suganthan, P.N., Coello, C.A.C., Herrera, F. (2021). A tutorial on the design, experimentation and application of metaheuristic algorithms to real-world optimization problems. Swarm and Evolutionary Computation, 64, 100888.
 
Oztas, G.Z., Erdem, S. (2021). Framework selection for developing optimization algorithms: assessing preferences by conjoint analysis and best–worst method. Soft Computing, 25(5), 3831–3848.
 
Parejo, J.A., Ruiz-Cortés, A., Lozano, S., Fernandez, P. (2012). Metaheuristic optimization frameworks: a survey and benchmarking. Soft Computing, 16, 527–561.
 
Ramírez, A., Barbudo, R., Romero, J.R. (2023). An experimental comparison of metaheuristic frameworks for multi-objective optimization. Expert Systems, 40(4), 12672.
 
Rapin, J., Teytaud, O. (2018). Nevergrad – A gradient-free optimization platform. GitHub.
 
Ravber, M., Mernik, M., Črepinšek, M. (2016). The impact of quality indicators on the rating of multi-objective evolutionary algorithms. In: Conference on Bioinspired Optimization Methods and their Applications (BIOMA 2016), pp. 119–130.
 
Ravber, M., Moravec, M., Mernik, M. (2022a). Primerjava evolucijskih algoritmov implementiranih v različnih programskih jezikih. Electrotechnical Review/Elektrotehniski Vestnik, 89, (1–2), 46–52.
 
Ravber, M., Liu, S.-H., Mernik, M., Črepinšek, M. (2022b). Maximum number of generations as a stopping criterion considered harmful. Applied Soft Computing, 128, 109478.
 
Riza, L.S., Iip, Nugroho, E.P., Munir (2018). MetaheuristicOpt: an R Package for optimisation based on meta-heuristics algorithms. Pertanika Journal of Science and Technology, 26(3), 1401–1411.
 
Sahin, O., Akay, B. (2016). Comparisons of metaheuristic algorithms and fitness functions on software test data generation. Applied Soft Computing, 49, 1202–1214. https://doi.org/10.1016/j.asoc.2016.09.045. https://www.sciencedirect.com/science/article/pii/S156849461630504X.
 
Shami, T.M., El-Saleh, A.A., Alswaitti, M., Al-Tashi, Q., Summakieh, M.A., Mirjalili, S. (2022). Particle swarm optimization: a comprehensive survey. IEEE Access, 10, 10031–10061.
 
Shi, Y., Eberhart, R. (1998). A modified particle swarm optimizer. In: 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No. 98TH8360). IEEE, pp. 69–73.
 
Silva, M.A.L., de Souza, S.R., Souza, M.J.F., de Franca Filho, M.F. (2018). Hybrid metaheuristics and multi-agent systems for solving optimization problems: a review of frameworks and a comparative analysis. Applied Soft Computing, 71, 433–459.
 
Storn, R., Price, K. (1997). Differential evolution–a simple and efficient heuristic for global optimization over continuous spaces. Journal of Global Optimization, 11, 341–359.
 
Tian, Y., Cheng, R., Zhang, X., Jin, Y. (2017). PlatEMO: a MATLAB platform for evolutionary multi-objective optimization. IEEE Computational Intelligence Magazine, 12(4), 73–87.
 
Tzanetos, A., Dounias, G. (2020). A comprehensive survey on the applications of swarm intelligence and bio-inspired evolutionary strategies. In: Machine Learning Paradigms: Advances in Deep Learning-Based Technological Applications, pp. 337–378.
 
Van Thieu, N., Mirjalili, S. (2023). MEALPY: an open-source library for latest meta-heuristic algorithms in Python. Journal of Systems Architecture, 139, 102871. https://doi.org/10.1016/j.sysarc.2023.102871.
 
Veček, N., Črepinšek, M., Mernik, M. (2017). On the influence of the number of algorithms, problems, and independent runs in the comparison of evolutionary algorithms. Applied Soft Computing, 54, 23–45.
 
Veček, N., Mernik, M., Črepinšek, M. (2014). A chess rating system for evolutionary algorithms: a new method for the comparison and ranking of evolutionary algorithms. Information Sciences, 277, 656–679.
 
Velasco, L., Guerrero, H., Hospitaler, A. (2024). A literature review and critical analysis of metaheuristics recently developed. Archives of Computational Methods in Engineering, 31(1), 125–146.
 
Villalobos, I., Ferrer, J., Alba, E. (2018). Measuring the quality of machine learning and optimization frameworks. In: Advances in Artificial Intelligence: 18th Conference of the Spanish Association for Artificial Intelligence, CAEPIA 2018, Granada, Spain, October 23–26, 2018, Proceedings 18. Springer, pp. 128–139.
 
Vrbančič, G., Brezočnik, L., Mlakar, U., Fister, D., Fister Jr., I. (2018). NiaPy: Python microframework for building nature-inspired algorithms. Journal of Open Source Software, 3(23), 613. https://doi.org/10.21105/joss.00613.
 
Wang, C.-H., Hu, K., Wu, X., Ou, Y. (2025). Rethinking metaheuristics: unveiling the Myth of “Novelty” in metaheuristic algorithms. Mathematics, 13(13), 2158.
 
Wang, Z., Qin, C., Wan, B., Song, W.W. (2021). A comparative study of common nature-inspired algorithms for continuous function optimization. Entropy, 23(7), 874.
 
Zelinka, I., Chadli, M., Davendra, D., Senkerik, R., Pluhacek, M., Lampinen, J. (2013). Do evolutionary algorithms indeed require random numbers? Extended study. In: Nostradamus 2013: Prediction, Modeling and Analysis of Complex Systems. Springer, pp. 61–75.

Biographies

Ravber Miha
https://orcid.org/0000-0003-4908-4631
miha.ravber@um.si

M. Ravber received his BSc, MSc, and PhD in computer science from the University of Maribor, Maribor, Slovenia, in 2012, 2015, and 2018, respectively. He is currently an assistant professor at the Faculty of Electrical Engineering and Computer Science, University of Maribor, Maribor, Slovenia. He has worked in the Programming Methodologies Laboratory since 2015. His research interests include evolutionary computation, single- and multi-objective optimization, soft computing, and genetic programming.

Šmid Marko
https://orcid.org/0009-0001-9099-6416
marko.smid2@um.si

M. Šmid received his BSc and MSc degrees in computer science from the University of Maribor, Maribor, Slovenia, in 2020 and 2022, respectively. He is currently pursuing a PhD in computer science and works as a technical assistant at the Faculty of Electrical Engineering and Computer Science, University of Maribor. Since 2022, he has been a member of the Programming Methodologies Laboratory. His research interests include evolutionary computation, genetic algorithms, genetic programming, multi-agent systems, and self-organizing systems.

Moravec Matej
https://orcid.org/0000-0003-0756-3300
matej.moravec@um.si

M. Moravec received his BSc and MSc degrees in computer science from the University of Maribor, Maribor, Slovenia, in 2017 and 2019, respectively. He is currently pursuing a PhD in computer science and works as a teaching assistant at the Faculty of Electrical Engineering and Computer Science, University of Maribor. Since 2019, he has been a member of the Programming Methodologies Laboratory. His research interests include evolutionary computation and single- and multi-objective dynamic optimization.

Mernik Marjan
https://orcid.org/0000-0002-2775-0667
marjan.mernik@um.si

M. Mernik received the MSc and PhD degrees in computer science from the University of Maribor in 1994 and 1998, respectively. He is currently a professor at the University of Maribor, Faculty of Electrical Engineering and Computer Science. He was a visiting professor at the University of Alabama at Birmingham, Department of Computer and Information Sciences. His research interests include programming languages, domain-specific (modelling) languages, grammar and semantic inference, and evolutionary computations. He is the editor-in-chief of the Journal of Computer Languages, as well as associate editors of the Applied Soft Computing Journal, and Swarm and Evolutionary Computation Journal. He has been named a Highly Cited Researcher for years 2017 and 2018. More information about his work is available at https://lpm.feri.um.si/en/members/mernik/.

Črepinšek Matej
https://orcid.org/0000-0003-2802-316X
matej.crepinsek@um.si

M. Črepinšek earned his BSc (1999) and PhD (2007) in computer science from the University of Maribor, Slovenia. He currently serves as an associate professor at the Faculty of Electrical Engineering and Computer Science, University of Maribor. His research interests span game development, mobile development, grammar inference, evolutionary computation, single- and multi-objective optimization, as well as computer science education.


Full article Related articles PDF XML
Full article Related articles PDF XML

Copyright
© 2025 Vilnius University
Open access article under the CC BY license.

Keywords
metaheuristics evolutionary algorithms metaheuristic optimization framework algorithm comparison benchmarking

Metrics
since January 2020
422

Article info
views

399

Full article
views

388

PDF
downloads

250

XML
downloads


Share


RSS