Understanding and Detecting Android R8 Compiler Bugs

Abstract

Android apps currently dominate the smartphone app market, and their reliability will significantly affect user experience and even induce security risks. Since Android Gradle Plugin 3.4.0 (April 2019), the R8 compiler serves as the default infrastructure in the Android build process. It transforms JVM bytecode from the Java layer into semantic-equivalent DEX bytecode, ensuring compatibility with the Android Virtual Machine. Additionally, R8 incorporates advanced features like shrinking and obfuscation to reduce app size and enhance security. However, R8 suffers from substantial bugs, some of which can cause severe issues. This motivates us to conduct a preliminary study into R8 bugs. We collect and analyze 945 bug reports for R8. Through detailed statistical analysis, we obtain several valuable findings. For example, we identify Optimization as the most error-prone component. To substantiate our findings, we develop an automated testing tool named R8Scan. It utilizes a novel idea to synthesize seeds from prioritized real-world functions and construct the corresponding arguments empowered by Large Language Models (LLMs) to test R8, thus enabling the exploration of a broader range of semantics. Finally, R8Scan detects 17 R8 bugs. It also detects 10 bugs in OpenJDK and 6 bugs in Android Runtime. Among the R8 bugs, 11 are assigned priority P1, the highest developer-assigned priority level. Extensive experiments demonstrate the superiority of R8Scan over state-of-the-art JVM fuzzers. We believe this study is valuable to enhance the security of the R8 compiler.