Android CTS 7.0_r1 : Running modules list through automated script

Hello Guys,
How are you doing? hope all is well. Its been while, lets get started!

So whats new? Yeah Android 7 is new! Android 7 has been out now for a while and everybody (device vendors) wants it up and running as soon as possible. So here I am trying to get Android 7 up for some super secrete project, just kidding, yes I (and my team) had just completed bring up of the device with Android N (i.e. Android 7, called Nougat) yeeeh! but that's just the half battle, still need to run Android Compatibility Test Suite (CTS) to win the rest.

So, here is the issue, The device I have has some serious battery problem, it cant charge through USB if its connected to computer, it needs to be connect to charger to do so. So every time I kick CTS plan device goes down before it completes. The solution is instead of running whole plan run module by module and get result but its heck lot of manual work and I have to keep eye on it. Also in CTS 7 google has changed structure a bit lot.

So here I am going to give you scripts which helps you run CTS modules one-by-one and also saves results location in log file. So if device goes down while running some module you can continue from where you left. You just need to follow some steps as bellow:

1. Download CTS 7  

You can download CTS package from : CTS download   and extract it.

 

2. Get list of modules

After extracting CTS package go to android-cts/tools directory and execute following command to get available modules list,

	cd android-cts/tools/
	./cts-tradefed l m

view raw cts-modules-list.sh hosted with ❤ by GitHub

Then copy list of modules to a file e.g. modules.lst or you can get it here 

	CtsAadbHostTestCases
	CtsAbiOverrideHostTestCases
	CtsAccelerationTestCases
	CtsAccessibilityServiceTestCases
	CtsAccessibilityTestCases
	CtsAccountManagerTestCases
	CtsAdminTestCases
	CtsAlarmClockTestCases
	CtsAndroidAppTestCases
	CtsAnimationTestCases
	CtsAppSecurityHostTestCases
	CtsAppTestCases
	CtsAppUsageHostTestCases
	CtsAppWidgetTestCases
	CtsAslrMallocTestCases
	CtsAssistTestCases
	CtsAtraceHostTestCases
	CtsBackupTestCases
	CtsBionicTestCases
	CtsBluetoothTestCases
	CtsCalendarcommon2TestCases
	CtsCallLogTestCases
	CtsCameraTestCases
	CtsCarTestCases
	CtsCarrierApiTestCases
	CtsCompilationTestCases
	CtsContentTestCases
	CtsCppToolsTestCases
	CtsDatabaseTestCases
	CtsDeqpTestCases
	CtsDeviceAdminUninstallerTestCases
	CtsDevicePolicyManagerTestCases
	CtsDisplayTestCases
	CtsDpiTestCases
	CtsDpiTestCases2
	CtsDragAndDropHostTestCases
	CtsDramTestCases
	CtsDreamsTestCases
	CtsDrmTestCases
	CtsDumpsysHostTestCases
	CtsEffectTestCases
	CtsExternalServiceTestCases
	CtsFileSystemTestCases
	CtsFragmentTestCases
	CtsGestureTestCases
	CtsGraphics2TestCases
	CtsGraphicsTestCases
	CtsHardwareTestCases
	CtsHostsideNetworkTests
	CtsHostsideNumberBlockingTestCases
	CtsHostsideTvTests
	CtsIcuTestCases
	CtsJankDeviceTestCases
	CtsJdwpSecurityHostTestCases
	CtsJdwpTestCases
	CtsJniTestCases
	CtsJobSchedulerTestCases
	CtsKeystoreTestCases
	CtsLeanbackJankTestCases
	CtsLibcoreLegacy22TestCases
	CtsLibcoreOjTestCases
	CtsLibcoreOkHttpTestCases
	CtsLibcoreTestCases
	CtsLocation2TestCases
	CtsLocationTestCases
	CtsMediaStressTestCases
	CtsMediaTestCases
	CtsMidiTestCases
	CtsMonkeyTestCases
	CtsMultiUserHostTestCases
	CtsMultiUserTestCases
	CtsNativeMediaSlTestCases
	CtsNativeMediaXaTestCases
	CtsNdefTestCases
	CtsNetSecConfigAttributeTestCases
	CtsNetSecConfigBasicDebugDisabledTestCases
	CtsNetSecConfigBasicDebugEnabledTestCases
	CtsNetSecConfigBasicDomainConfigTestCases
	CtsNetSecConfigCleartextTrafficTestCases
	CtsNetSecConfigInvalidPinTestCases
	CtsNetSecConfigNestedDomainConfigTestCases
	CtsNetSecConfigResourcesSrcTestCases
	CtsNetSecPolicyUsesCleartextTrafficFalseTestCases
	CtsNetSecPolicyUsesCleartextTrafficTrueTestCases
	CtsNetSecPolicyUsesCleartextTrafficUnspecifiedTestCases
	CtsNetTestCases
	CtsNetTestCasesLegacyApi22
	CtsNetTestCasesLegacyPermission22
	CtsOpenGLTestCases
	CtsOpenGlPerf2TestCases
	CtsOpenGlPerfTestCases
	CtsOsHostTestCases
	CtsOsTestCases
	CtsPermission2TestCases
	CtsPermissionTestCases
	CtsPreference2TestCases
	CtsPreferenceTestCases
	CtsPrintTestCases
	CtsProviderTestCases
	CtsRenderscriptLegacyTestCases
	CtsRenderscriptTestCases
	CtsRsBlasTestCases
	CtsRsCppTestCases
	CtsSampleDeviceTestCases
	CtsSampleHostTestCases
	CtsSaxTestCases
	CtsSecurityHostTestCases
	CtsSecurityTestCases
	CtsServicesHostTestCases
	CtsSignatureTestCases
	CtsSimpleCpuTestCases
	CtsSimpleperfTestCases
	CtsSpeechTestCases
	CtsSustainedPerformanceHostTestCases
	CtsSystemUiHostTestCases
	CtsSystemUiTestCases
	CtsTelecomTestCases
	CtsTelecomTestCases2
	CtsTelephony2TestCases
	CtsTelephonyTestCases
	CtsTextTestCases
	CtsTextureViewTestCases
	CtsThemeDeviceTestCases
	CtsThemeHostTestCases
	CtsTransitionTestCases
	CtsTrustedVoiceHostTestCases
	CtsTvProviderTestCases
	CtsTvTestCases
	CtsUiAutomationTestCases
	CtsUiDeviceTestCases
	CtsUiHostTestCases
	CtsUiRenderingTestCases
	CtsUidIsolationTestCases
	CtsUsageStatsTestCases
	CtsUsbTests
	CtsUtilTestCases
	CtsVideoTestCases
	CtsViewTestCases
	CtsVmTestCases
	CtsVoiceInteractionTestCases
	CtsVoiceSettingsTestCases
	CtsVrTestCases
	CtsWebkitTestCases
	CtsWidgetTestCases
	vm-tests-tf

view raw modules.lst hosted with ❤ by GitHub

3. Script to run modules one-by-one from the list

Now what you need is a script that runs modules from above list one-by-one, here it is, 

	#! /bin/bash
	# Execute CTS modules listed in file $1 one after another
	# https://thinking-electron.blogspot.com/
	BASE_CMD="./cts-tradefed run cts --skip-preconditions --disable-reboot --skip-device-info -m "
	TESTS_FILE=$1
	TESTS_LOG=$TESTS_FILE.log
	trap "exit" INT
	while read line
	do
	TM=$(date +'%d%m%y-%T')
	echo -e "*[$TM]@TE: Starting test - $BASE_CMD$line"
	echo "*[$TM]@TE: Starting test - $BASE_CMD$line" >> $TESTS_LOG
	echo exit | $BASE_CMD$line
	TM2=$(date +'%d%m%y-%T')
	echo "*[$TM2]@TE: Finished test - $BASE_CMD$line" >> $TESTS_LOG
	done <$TESTS_FILE

view raw run-cts-modules.sh hosted with ❤ by GitHub

Running the CTS modules using script,

./run-cts-mosules.sh modules.lst

view raw runnung-the-cts-script.sh hosted with ❤ by GitHub

If device dies while running some module using the script, you can press 'ctrl+c' to stop script in between and remove modules from 'modules.lst' file which ran to completion successfully, then run the script again as before to continue from where you left.

4. Collecting logs of CTS

Result of each module ran using above script can be found in android-cts/results/ directory in separate sub-directories with sub-directory name as time-stamp when respective module has started. Above script collects all modules time-stamp in log file, modules.lst.log, which you can use latter to correlate time-stamps recorded to find each modules result/log directory. See snap-shot bellow,

	...
	*[190916-18:39:07]@TE: Starting test - ./cts-tradefed run cts --skip-preconditions --disable-reboot --skip-device-info -m CtsEffectTestCases
	*[190916-18:39:27]@TE: Finished test - ./cts-tradefed run cts --skip-preconditions --disable-reboot --skip-device-info -m CtsEffectTestCases
	*[190916-18:39:27]@TE: Starting test - ./cts-tradefed run cts --skip-preconditions --disable-reboot --skip-device-info -m CtsExternalServiceTestCases
	*[190916-18:39:55]@TE: Finished test - ./cts-tradefed run cts --skip-preconditions --disable-reboot --skip-device-info -m CtsExternalServiceTestCases
	...

view raw modules.lst.log hosted with ❤ by GitHub

That's it guys, have a nice day!!

Cheers!!!