Write a Blog >>
ESEC/FSE 2021
Thu 19 - Sat 28 August 2021 Clowdr Platform
Thu 26 Aug 2021 12:00 - 12:10 - Human Aspects—HCI and Mobile Chair(s): Jürgen Cito
Fri 27 Aug 2021 00:00 - 00:10 - Human Aspects—HCI and Mobile Chair(s): Gustavo Pinto

Mobile apps are playing an increasingly important role in our daily lives, including the lives of approximately 304 million users worldwide that are either completely blind or suffer from some form of visual impairment. These users rely on screen readers to interact with apps. Screen readers, however, cannot describe the image icons that appear on the screen, unless those icons are accompanied with developer-provided textual labels. A prior study of over 5,000 Android apps found that in around 50% of the apps, less than 10% of the icons are labeled. To address this problem, a recent award-winning approach, called LabelDroid, employed deep-learning techniques to train a model on a dataset of existing icons with labels to automatically generate labels for visually similar, unlabeled icons.
In this work, we empirically study the nature of icon labels in terms of distribution and their dependency on different sources of information. We then assess the effectiveness of LabelDroid in predicting labels for unlabeled icons. We find that icon images are insufficient in representing icon labels, while other sources of information from the icon usage context can enrich images in determining proper tokens for labels. We propose the first context-aware label generation approach, called COALA, that incorporates several sources of information from the icon in generating accurate labels. Our experiments show that although COALA significantly outperforms LabelDroid in both user study and automatic evaluation, further research is needed. We suggest that future studies should be more cautious when basing their approach on automatically extracted labeled data.

Thu 26 Aug

Displayed time zone: Athens change

12:00 - 13:00
Human Aspects—HCI and MobileResearch Papers / Industry Papers +12h
Chair(s): Jürgen Cito TU Vienna; Facebook
12:00
10m
Paper
Data-Driven Accessibility Repair Revisited: On the Effectiveness of Generating Labels for Icons in Android Apps
Research Papers
Forough Mehralian University of California at Irvine, Navid Salehnamadi University of California at Irvine, Sam Malek University of California at Irvine
DOI
12:10
10m
Paper
Benchmarking Automated GUI Testing for Android against Real-World BugsArtifacts AvailableArtifacts Reusable
Research Papers
Ting Su East China Normal University, Jue Wang Nanjing University, Zhendong Su ETH Zurich
DOI Pre-print Media Attached
12:20
10m
Paper
An Empirical Study of GUI Widget Detection for Industrial Mobile Games
Industry Papers
Jiaming Ye Kyushu University, Ke Chen Fuxi AI Lab of Netease, Xiaofei Xie Kyushu University, Lei Ma University of Alberta, Ruochen Huang University of Alberta, Yingfeng Chen Fuxi AI Lab of Netease, Yinxing Xue University of Science and Technology of China, Jianjun Zhao Kyushu University
DOI
12:30
30m
Live Q&A
Q&A (Human Aspects—HCI and Mobile)
Research Papers

Fri 27 Aug

Displayed time zone: Athens change

00:00 - 01:00
Human Aspects—HCI and MobileResearch Papers / Industry Papers
Chair(s): Gustavo Pinto Federal University of Pará (UFPA) and Zup Innovation
00:00
10m
Paper
Data-Driven Accessibility Repair Revisited: On the Effectiveness of Generating Labels for Icons in Android Apps
Research Papers
Forough Mehralian University of California at Irvine, Navid Salehnamadi University of California at Irvine, Sam Malek University of California at Irvine
DOI
00:10
10m
Paper
Benchmarking Automated GUI Testing for Android against Real-World BugsArtifacts AvailableArtifacts Reusable
Research Papers
Ting Su East China Normal University, Jue Wang Nanjing University, Zhendong Su ETH Zurich
DOI Pre-print Media Attached
00:20
10m
Paper
An Empirical Study of GUI Widget Detection for Industrial Mobile Games
Industry Papers
Jiaming Ye Kyushu University, Ke Chen Fuxi AI Lab of Netease, Xiaofei Xie Kyushu University, Lei Ma University of Alberta, Ruochen Huang University of Alberta, Yingfeng Chen Fuxi AI Lab of Netease, Yinxing Xue University of Science and Technology of China, Jianjun Zhao Kyushu University
DOI
00:30
30m
Live Q&A
Q&A (Human Aspects—HCI and Mobile)
Research Papers