Android Day 2: Fastlane

[Abhiek187]

I wrote unit tests for RecipeRepository and MainViewModel using JUnit 5 (also known as Jupiter). Even though Android Studio defaults to JUnit 4, I wanted to give the newer version a shot. I had to conjure some interesting mocks to get the unit tests to work while in coroutines (Kotlin's equivalent of a thread). Alongside that, I also initialized Fastlane and wrote a workflow to automate all the testing. Unlike with iOS, the gradle test command only runs the unit tests. I had some issues testing Fastlane on my PC (plus I think the network bridge between the WSL 2 VM and my C drive is causing gradle to be even slower on the command line), so I'm going to run the workflow here to test it out.

For the instrumentation tests, I have two options: use the GitHub Action like in calculators, or use the Fastlane plugin. I'm going to try the Fastlane plugin to see how that compares to the GitHub Action. I like how customizable the action is, but if this is faster, I might go with the plugin instead. But since the iOS workflows were already slow, I'm curious to see how long these workflows will take. (I also wonder if there will be any problems running on a Mac since Gemfile.lock has the platform set to x86_64-linux, instead of universal-darwin-21.)

[Abhiek187]

This has been an interesting afternoon. So I found two Fastlane plugins to help automate these instrumentation tests: fastlane-plugin-instrumented_tests and fastlane-plugin-automated-test-emulator-run. However, I was struggling to figure out why these weren't working. Both were throwing the same error:

Error: Exception in thread "main" java.lang.NoClassDefFoundError: javax/xml/bind/annotation/XmlSchema
	at com.android.repository.api.SchemaModule$SchemaModuleVersion.<init>(SchemaModule.java:156)
	at com.android.repository.api.SchemaModule.<init>(SchemaModule.java:75)
	at com.android.sdklib.repository.AndroidSdkHandler.<clinit>(AndroidSdkHandler.java:81)
	at com.android.sdklib.tool.AvdManagerCli.run(AvdManagerCli.java:213)
	at com.android.sdklib.tool.AvdManagerCli.main(AvdManagerCli.java:200)
Caused by: java.lang.ClassNotFoundException: javax.xml.bind.annotation.XmlSchema
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:[52](https://github.com/Abhiek187/ez-recipes-android/actions/runs/3401672022/jobs/5656938365#step:7:53)2)
	... 5 more

In the process, I documented what commands are run under the hood when we try to run the unit and UI tests from a CI server, for both iOS and Android. On iOS, since Xcode comes with many simulators, the main challenge is passing in the right parameters in a long xcodebuild command to build and test the app. On Android, although unit testing is as simple as running gradlew test, for UI tests we need to install the build tools, platform tools, system images, and emulators. And we need to boot the emulators ourselves before running gradlew connectedAndroidTest (or connectedCheck since they seem to be similar). That would explain why the Android tests in the calculators repo take longer than the iOS tests.

Finally, I figured out the reason I was getting the above error is because both plugins execute Android's command line tools from $ANDROID_HOME/tools/bin. However, as of 2020, the command line tools now point to $ANDROID_HOME/cmdline-tools/latest/bin. The command still exists in the tools directory but doesn't work properly. I experienced something similar before when I was trying to run the emulator from the command line, so I understand why I was confused. 😅

Someone did create a PR in fastlane-plugin-automated-test-emulator-run to address this issue, but it seems like the repo isn't actively maintained anymore (given that the plugin was initially created in 2016). I do want to try the person's fork since I still want to compare performance, but since I'm using GitHub Actions, I think android-emulator-runner might be my best option for the long-term. But knowing the exact commands to run will still help if I have to replicate this outside GitHub.

[Abhiek187]

After more experimentation, here's what I found. The instrumentation tests seem to work best if we delegate one device per job. I kept trying to run the tests on three devices at once and I kept getting errors similar to the following:

Test run failed to complete. No test results. onError: commandError=false message=INSTRUMENTATION_ABORTED: System has crashed.
Exception thrown during onBeforeAll invocation of plugin com.google.testing.platform.plugin.android.AndroidDevicePlugin.
Failed to install APK(s): /Users/runner/work/ez-recipes-android/ez-recipes-android/EZRecipes/app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk
Unknown failure: Exception occurred while executing 'install':
java.lang.NullPointerException: Attempt to invoke virtual method 'java.util.List android.os.storage.StorageManager.getVolumes()' on a null object reference
at com.android.internal.content.PackageHelper.resolveInstallVolume(PackageHelper.java:197)
at com.android.internal.content.PackageHelper.resolveInstallVolume(PackageHelper.java:147)
at com.android.internal.content.PackageHelper.resolveInstallVolume(PackageHelper.java:162)
at com.android.server.pm.PackageInstallerService.createSessionInternal(PackageInstallerService.java:776)
at com.android.server.pm.PackageInstallerService.createSession(PackageInstallerService.java:561)
at com.android.server.pm.PackageManagerShellCommand.doCreateSession(PackageManagerShellCommand.java:3143)
at com.android.server.pm.PackageManagerShellCommand.doRunInstall(PackageManagerShellCommand.java:1341)
at com.android.server.pm.PackageManagerShellCommand.runInstall(PackageManagerShellCommand.java:1303)
at com.android.server.pm.PackageManagerShellCommand.onCommand(PackageManagerShellCommand.java:193)
at com.android.modules.utils.BasicShellCommandHandler.exec(BasicShellCommandHandler.java:97)
at android.os.ShellCommand.exec(ShellCommand.java:38)
at com.android.server.pm.PackageManagerService.onShellCommand(PackageManagerService.java:24626)
at android.os.Binder.shellCommand(Binder.java:950)
at android.os.Binder.onTransact(Binder.java:834)
at android.content.pm.IPackageManager$Stub.onTransact(IPackageManager.java:4818)

The problem is that java.lang.NullPointerException: Attempt to invoke virtual method... is such a generic error that it was hard to find people who get the same error and are also trying to run Android tests. I don't get this error if I run the instrumentation tests locally. I then tried to use android-emulator-runner like in calculators and the tests were finally passing! In that repo, I split each API into its own job, which means that I don't need to loop through all the devices in series to make sure they're booted and ready to test. I was planning on parallelizing the tests anyway to improve performance, but I didn't realize that was the only way to ensure the tests would pass.

Now the issue is that if I want to continue using Fastlane, I would need to pass the matrix variables so that each lane would only run the tests on a device with that specific API level. According to the docs, I can pass in an options parameter to pass in custom parameters for each lane. But if I were to use fastlane-plugin-automated-test-emulator-run, I would need to create separate JSON files for each API level, which wouldn't scale well and would be tougher to maintain. I could also try the manual sh commands (since that worked miraculously). Both the manual commands and the GitHub Action have similar performance (testing on an Android 29 emulator took around 6.5 minutes for both, with the Action being slightly faster). I'll likely stick with the Action since that will be the easiest to maintain, but I'd like to preserve the manual commands in the commit history to show that it is possible.

[Abhiek187]

Job	Fastlane	GitHub Actions
29	6m 56s	4m 38s
31	8m 34s	6m 36s
33	13m 28s	8m 34s

GitHub Actions beats Fastlane for all API levels. Looking at each step individually (for API 33):

Step	Fastlane	GitHub Actions
Accept licenses	7s	4s
Install build/platform tools	3s	3s
Install emulator	3s	2s
Install system images	53s	1m 59s
Create AVD	1s	2s
Configure 2 cores	0s	0s
Start emulator	0s	10s
Wait to boot	5m 51s	3m 24s
Disable animations	1s	1s
Run tests	6m 24s	2m 48s
Kill emulator	0s	0s

Installing the system images, booting up the emulator, and running the UI tests take the slowest, and show the most significant performance differences between the two methods. One thing to consider is that I added the unit test step in GitHub Actions, which allowed the UI tests step to run quicker since the Gradle daemon was running and some of the tasks were already up-to-date. But that's still 2.5 minutes unaccounted for Fastlane. It might be because the compileDebugKotlin task took almost 2 minutes for Fastlane, whereas it took 2 seconds for GA. I'm not sure why. But besides that, I didn't notice any other significant differences between the two methods. But this was a nice analysis to understand the world of Android! 👍

And as a bonus, when comparing the different API versions, the most significant difference was the boot time: 1m 27s for API 29, 3m 6s for API 31, and 5m 50s for API 33.

[Abhiek187]

Hopefully this will be my last commit for this PR! 🤞 I noticed that as long as I comment out the Fastlane steps in the workflow, I can keep the manual code I used in the Fastfile for reference.