On Comparing Mutation Testing Tools through Learning-based Mutant Selection

Recently many mutation testing tools have been proposed that rely on bug-fix patterns and natural language models trained on large code corpus. As these tools operate fundamentally differently from the grammar-based traditional approaches, a question arises of how these tools compare in terms of 1)...

Descripción completa

Guardado en:
Detalles Bibliográficos
Autores principales: Ojdanic, Milos, Khanfir, Ahmed, Garg, Aayush, Degiovanni, Renzo, Papadakis, Mike, Le Traon, Yves
Formato: Objeto de conferencia Resumen
Lenguaje:Inglés
Publicado: 2023
Materias:
Ciencias Informáticas
Acceso en línea:http://sedici.unlp.edu.ar/handle/10915/165815
Aporte de:
SEDICI (UNLP) de Universidad Nacional de La Plata
id I19-R120-10915-165815
record_format dspace
spelling I19-R120-10915-1658152024-05-09T20:05:05Z http://sedici.unlp.edu.ar/handle/10915/165815 On Comparing Mutation Testing Tools through Learning-based Mutant Selection Ojdanic, Milos Khanfir, Ahmed Garg, Aayush Degiovanni, Renzo Papadakis, Mike Le Traon, Yves 2023-09 2023 2024-05-09T12:37:59Z en Ciencias Informáticas Recently many mutation testing tools have been proposed that rely on bug-fix patterns and natural language models trained on large code corpus. As these tools operate fundamentally differently from the grammar-based traditional approaches, a question arises of how these tools compare in terms of 1) fault detection and 2) cost-effectiveness. Simultaneously, mutation testing research proposes mutant selection approaches based on machine learning to mitigate its applica- tion cost. This raises another question: How do the existing mutation testing tools compare when guided by mutant selection approaches? To answer these questions, we compare four existing tools – μBERT (uses pre-trained language model for fault seeding), IBIR (relies on inverted fix-patterns), DeepMutation (generates mutants by employing Neural Machine Translation) and PIT (applies standard grammar-based rules) in terms of fault detection capability and cost-effectiveness, in conjunction with standard and deep learning based mutant selection strategies. Our results show that IBIR has the highest fault detection capability among the four tools; however, it is not the most cost-effective when considering different selection strategies. On the other hand, μBERT having a relatively lower fault detection capability, is the most cost-effective among the four tools. Our results also indicate that comparing mutation testing tools when using deep learning-based mutant selection strategies can lead to different conclusions than the standard mutant selection. For instance, our results demonstrate that combining μBERT with deep learning-based mutant selection yields 12% higher fault detection than the considered tools. Sociedad Argentina de Informática e Investigación Operativa Objeto de conferencia Resumen http://creativecommons.org/licenses/by-nc-sa/4.0/ Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) application/pdf 72-72
institution Universidad Nacional de La Plata
institution_str I-19
repository_str R-120
collection SEDICI (UNLP)
language Inglés
topic Ciencias Informáticas
spellingShingle Ciencias Informáticas
Ojdanic, Milos
Khanfir, Ahmed
Garg, Aayush
Degiovanni, Renzo
Papadakis, Mike
Le Traon, Yves
On Comparing Mutation Testing Tools through Learning-based Mutant Selection
topic_facet Ciencias Informáticas
description Recently many mutation testing tools have been proposed that rely on bug-fix patterns and natural language models trained on large code corpus. As these tools operate fundamentally differently from the grammar-based traditional approaches, a question arises of how these tools compare in terms of 1) fault detection and 2) cost-effectiveness. Simultaneously, mutation testing research proposes mutant selection approaches based on machine learning to mitigate its applica- tion cost. This raises another question: How do the existing mutation testing tools compare when guided by mutant selection approaches? To answer these questions, we compare four existing tools – μBERT (uses pre-trained language model for fault seeding), IBIR (relies on inverted fix-patterns), DeepMutation (generates mutants by employing Neural Machine Translation) and PIT (applies standard grammar-based rules) in terms of fault detection capability and cost-effectiveness, in conjunction with standard and deep learning based mutant selection strategies. Our results show that IBIR has the highest fault detection capability among the four tools; however, it is not the most cost-effective when considering different selection strategies. On the other hand, μBERT having a relatively lower fault detection capability, is the most cost-effective among the four tools. Our results also indicate that comparing mutation testing tools when using deep learning-based mutant selection strategies can lead to different conclusions than the standard mutant selection. For instance, our results demonstrate that combining μBERT with deep learning-based mutant selection yields 12% higher fault detection than the considered tools.
format Objeto de conferencia
Resumen
author Ojdanic, Milos
Khanfir, Ahmed
Garg, Aayush
Degiovanni, Renzo
Papadakis, Mike
Le Traon, Yves
author_facet Ojdanic, Milos
Khanfir, Ahmed
Garg, Aayush
Degiovanni, Renzo
Papadakis, Mike
Le Traon, Yves
author_sort Ojdanic, Milos
title On Comparing Mutation Testing Tools through Learning-based Mutant Selection
title_short On Comparing Mutation Testing Tools through Learning-based Mutant Selection
title_full On Comparing Mutation Testing Tools through Learning-based Mutant Selection
title_fullStr On Comparing Mutation Testing Tools through Learning-based Mutant Selection
title_full_unstemmed On Comparing Mutation Testing Tools through Learning-based Mutant Selection
title_sort on comparing mutation testing tools through learning-based mutant selection
publishDate 2023
url http://sedici.unlp.edu.ar/handle/10915/165815
work_keys_str_mv AT ojdanicmilos oncomparingmutationtestingtoolsthroughlearningbasedmutantselection
AT khanfirahmed oncomparingmutationtestingtoolsthroughlearningbasedmutantselection
AT gargaayush oncomparingmutationtestingtoolsthroughlearningbasedmutantselection
AT degiovannirenzo oncomparingmutationtestingtoolsthroughlearningbasedmutantselection
AT papadakismike oncomparingmutationtestingtoolsthroughlearningbasedmutantselection
AT letraonyves oncomparingmutationtestingtoolsthroughlearningbasedmutantselection
_version_ 1807223102187241472

Ejemplares similares