Write a Blog >>
ESEC/FSE 2021
Thu 19 - Sat 28 August 2021 Clowdr Platform

Testing of Deep Learning (DL) models is difficult due to the lack of automated test oracle and the high cost of human labelling. Differential testing has been used as a surrogate oracle, but there is no systematic guide on how to choose the reference model to use for differential testing.
We propose a novel differential testing approach based on subspecialized models, i.e., models that are trained on sliced training data only (hence specialized for the slice). A preliminary evaluation of our approach with an CNN-based EMNIST image classifier shows that it can achieve higher error detection rate with selected inputs compared to using more advanced ResNet and LeNet as the reference model for differential testing. Our approach also outperforms $N$-version testing, i.e., the use of the same DL model architecture trained separately but using the same data.

Wed 25 Aug

Displayed time zone: Athens change

16:00 - 17:00
SE & AI—Software Engineering for Machine Learning 2Research Papers / Journal First / Ideas, Visions and Reflections +12h
Chair(s): Matthew B Dwyer University of Virginia
16:00
10m
Paper
Fair Preprocessing: Towards Understanding Compositional Fairness of Data Transformers in Machine Learning PipelineArtifacts FunctionalArtifacts Available
Research Papers
Sumon Biswas Iowa State University, Hridesh Rajan Iowa State University
DOI Pre-print Media Attached
16:10
10m
Paper
Fairea: A Model Behaviour Mutation Approach to Benchmarking Bias Mitigation MethodsArtifacts FunctionalArtifacts Available
Research Papers
Max Hort University College London, Jie M. Zhang University College London, Federica Sarro University College London, Mark Harman University College London
DOI Pre-print
16:20
5m
Paper
Selecting Test Inputs for DNNs using Differential Testing with Subspecialized Model Instances
Ideas, Visions and Reflections
Yu-Seung Ma Electronics and Telecommunications Research Institute, Shin Yoo KAIST, Taeho Kim Electronics and Telecommunications Research Institute
DOI
16:25
5m
Paper
The Current State of Industrial Practice in Artificial Intelligence Ethics
Journal First
Ville Vakkuri University of Jyvaskyla, Kai-Kristian Kemell University of Jyvaskyla, Joni Kultanen University of Jyvaskyla, Pekka Abrahamsson University of Jyväskylä
16:30
30m
Live Q&A
Q&A (SE & AI—Software Engineering for Machine Learning 2)
Research Papers

Thu 26 Aug

Displayed time zone: Athens change

04:00 - 05:00
SE & AI—Software Engineering for Machine Learning 2Research Papers / Ideas, Visions and Reflections / Journal First
Chair(s): Tushar Sharma Siemens Research
04:00
10m
Paper
Fair Preprocessing: Towards Understanding Compositional Fairness of Data Transformers in Machine Learning PipelineArtifacts FunctionalArtifacts Available
Research Papers
Sumon Biswas Iowa State University, Hridesh Rajan Iowa State University
DOI Pre-print Media Attached
04:10
10m
Paper
Fairea: A Model Behaviour Mutation Approach to Benchmarking Bias Mitigation MethodsArtifacts FunctionalArtifacts Available
Research Papers
Max Hort University College London, Jie M. Zhang University College London, Federica Sarro University College London, Mark Harman University College London
DOI Pre-print
04:20
5m
Paper
Selecting Test Inputs for DNNs using Differential Testing with Subspecialized Model Instances
Ideas, Visions and Reflections
Yu-Seung Ma Electronics and Telecommunications Research Institute, Shin Yoo KAIST, Taeho Kim Electronics and Telecommunications Research Institute
DOI
04:25
5m
Paper
The Current State of Industrial Practice in Artificial Intelligence Ethics
Journal First
Ville Vakkuri University of Jyvaskyla, Kai-Kristian Kemell University of Jyvaskyla, Joni Kultanen University of Jyvaskyla, Pekka Abrahamsson University of Jyväskylä
04:30
30m
Live Q&A
Q&A (SE & AI—Software Engineering for Machine Learning 2)
Research Papers