SZZ误标变更对移动APP即时缺陷预测性能和解释的影响
作者:
中图分类号:

TP311

基金项目:

国家自然科学基金(61902228, 62176069, U23A20302); 陕西省自然科学基础研究计划(2024JC-YBMS-497); 陕西省重点研发计划 (2023-YBGY-265)


Impact of Mislabeled Changes by SZZ on Performance and Interpretation of Just-in-time Defect Prediction for Mobile APP
Author:
  • 摘要
  • | |
  • 访问统计
  • |
  • 参考文献
  • | |
  • 引证文献
  • | |
  • 文章评论
    摘要:

    近年来, SZZ作为一种识别引入缺陷的变更算法, 被广泛应用于即时软件缺陷预测技术中. 先前的研究表明, SZZ算法在对数据进行标注时会存在误标问题, 这将影响数据集的质量, 进而影响预测模型的性能. 因此, 研究人员对SZZ算法进行了改进, 并提出多个SZZ变体. 然而, 目前尚未有文献研究数据标注质量对移动APP即时缺陷预测性能和解释的影响. 为探究SZZ错误标注的变更对移动APP即时软件缺陷预测模型的影响, 对4种SZZ算法进行广泛而深入的实证研究. 首先, 选取GitHub库中17个大型移动APP项目, 借助PyDriller工具抽取软件度量元. 其次, 采用B-SZZ (原始SZZ版本)、AG-SZZ、MA-SZZ和RA-SZZ这4种算法标注数据. 然后, 根据时间序列划分数据, 利用随机森林、朴素贝叶斯和逻辑回归分类器分别建立即时缺陷预测模型. 最后, 使用AUCMCCG-mean传统指标和F-measure@20%、IFA工作量感知指标评估模型性能, 并使用SKESD和SHAP算法对结果进行统计显著性检验与可解释性分析. 通过对比4种SZZ算法的标注性能, 研究发现: (1) 数据的标注质量符合SZZ变体之间的递进关系; (2) B-SZZ、AG-SZZ 和MA-SZZ错误标注的变更会造成AUCMCC 得分不同程度的下降, 但不会造成G-mean得分下降; (3) B-SZZ会造成F-measure@20%得分下降, 而在代码审查时, B-SZZ、AG-SZZ 和MA-SZZ不会导致审查工作量的增加; (4)在模型解释方面, 不同SZZ算法会影响预测过程中贡献程度排名前3的度量元, 并且la度量元对预测结果有重要影响.

    Abstract:

    In recent years, as an algorithm for identifying bug-introducing changes, SZZ has been widely employed in just-in-time software defect prediction. Previous studies show that the SZZ algorithm may mislabel data during data annotation, which could influence the dataset quality and consequently the performance of the defect prediction model. Therefore, researchers have made improvements to the SZZ algorithm and proposed multiple variants of SZZ. However, there is no empirical study to explore the effect of data annotation quality by SZZ on the performance and interpretability of just-in-time defect prediction for mobile APP. To investigate the influence of mislabeled changes by SZZ on just-in-time defect prediction for mobile APP, this study conducts an extensive and in-depth empirical comparison of four SZZ algorithms. Firstly, 17 large-scale mobile APP projects are selected from the GitHub repository, and software metrics are extracted by adopting the PyDriller tool. Then, B-SZZ (original SZZ), AG-SZZ, MA-SZZ, and RA-SZZ are employed for data annotation. Then, the just-in-time defect prediction models are built with random forest, naive Bayes, and logistic regression classifiers based on the time-series data partitioning. Finally, the performance of the models is evaluated by traditional measures of AUC, MCC, and G-mean, and effort-aware measures of F-measure@20% and IFA, and a statistical significance test and interpretability analysis are conducted on the results by employing SKESD and SHAP respectively. By comparing the annotation performance of the four SZZ algorithms, the results are as follows. (1) The data annotation quality conforms to the progressive relationship among SZZ variants. (2) The mislabeled changes by B-SZZ, AG-SZZ, and MA-SZZ can cause performance reduction of AUC and MCC of different levels, but cannot lead to performance reduction of G-mean. (3) B-SZZ is likely to cause a performance reduction of F-measure@20%, while B-SZZ, AG-SZZ, and MA-SZZ are unlikely to increase effort during code inspection. (4) In terms of model interpretation, different SZZ algorithms will influence the three metrics with the largest contribution during the prediction, and the la metric has a significant influence on the prediction results.

    参考文献
    相似文献
    引证文献
    网友评论
    网友评论
    分享到微博
    发 布
引用本文

李志强,马睿,张洪宇,荆晓远,任杰,刘金会. SZZ误标变更对移动APP即时缺陷预测性能和解释的影响.软件学报,,():1-32

复制
分享
文章指标
  • 点击次数:51
  • 下载次数: 201
  • HTML阅读次数: 0
  • 引用次数: 0
历史
  • 收稿日期:2023-09-28
  • 最后修改日期:2023-12-26
  • 在线发布日期: 2025-02-19
文章二维码
您是第19728269位访问者
版权所有:中国科学院软件研究所 京ICP备05046678号-3
地址:北京市海淀区中关村南四街4号,邮政编码:100190
电话:010-62562563 传真:010-62562533 Email:jos@iscas.ac.cn
技术支持:北京勤云科技发展有限公司

京公网安备 11040202500063号