Add a Nova workflow to run iOS test on AWS Device Farm (#4973)

This adds a new Nova workflow called `mobile_job` that can be used to
run app tests on AWS Device Farm

1. In the initial version, the workflow will support running xctest
suite for iOS app. It requires the iOS IPA app archive and xctest suite
as inputs per
https://docs.aws.amazon.com/devicefarm/latest/developerguide/test-types-ios-xctest.html.
For example,
```
jobs:
  test-mobile-job:
    permissions:
      id-token: write
      contents: read
    uses: ./.github/workflows/mobile_job.yml
    with:
      device-type: ios
      runner: ubuntu-latest
      # This is PyTorch project on Device Farm
      project-arn: arn:aws:devicefarm:us-west-2:308535385114:project:b531574a-fb82-40ae-b687-8f0b81341ae0
      # The caller workflow will need to provide these artifacts. GHA support could be added later together with S3
      ios-ipa-archive: https://ossci-assets.s3.amazonaws.com/TestApp.ipa
      ios-xctest-zip: https://ossci-assets.s3.amazonaws.com/TestAppTests.xctest.zip
```

2. The main logic is in the script
`tools/device-farm-runner/run_on_aws_devicefarm.py` where it uploads the
IPA app archive and xctest suite to Device Farm, pools for the results,
and reports back. For example,

```
python run_on_aws_devicefarm.py --project-arn arn:aws:devicefarm:us-west-2:308535385114:project:b531574a-fb82-40ae-b687-8f0b81341ae0  --app-file ~/TestApp.ipa --xctest-file ~/TestAppTests.xctest.zip --name-prefix debug
```

3. The permission to Device Farm is granted by the role
`gha_workflow_mobile_job` from
pytorch-labs/pytorch-gha-infra#348

### Testing

On CI, I setup a test workflow to run PyTorch iOS app tests
https://github.com/pytorch/test-infra/actions/runs/8074168433/job/22059038480?pr=4973

Locally,

* If the run on Device Farm passes:
```
Run debug-invalid-workflow-id-0-2024-02-27-qVMFnLAf is scheduled as arn:aws:devicefarm:us-west-2:308535385114:run:b531574a-fb82-40ae-b687-8f0b81341ae0/295735c2-2a07-46f8-be67-ef9604a62d17
mobile-job-ios-8060493871-2-2024-02-27-IdKPJMzZ PASSED with stats {'total': 195, 'passed': 195, 'failed': 0, 'warned': 0, 'errored': 0, 'stopped': 0, 'skipped': 0}
```

* If the run on Device Farm fails, the failed tests will be print
together with the link to their artifacts on S3 for manual debugging:
```
Run debug-invalid-workflow-id-0-2024-02-27-XNivrOMS is scheduled as arn:aws:devicefarm:us-west-2:308535385114:run:b531574a-fb82-40ae-b687-8f0b81341ae0/fea59a70-c1ef-41ac-81c3-b8225b4aecf1
PyTorch-2023-09-29-OeFtuHCz FAILED
  Apple iPad Pro 12.9" (2022) PASSED
  Apple iPhone 11 PASSED
  Apple iPhone 13 FAILED
    Setup Suite PASSED with stats {'total': 1, 'passed': 1, 'failed': 0, 'warned': 0, 'errored': 0, 'stopped': 0, 'skipped': 0}
    TestAppTests FAILED with stats {'total': 37, 'passed': 36, 'failed': 1, 'warned': 0, 'errored': 0, 'stopped': 0, 'skipped': 0}
      testTorchScriptCollectionQuantOps PASSED
      testTorchScriptBuiltinQuantOps PASSED
      testFusedQuantOps PASSED
      testStaticQuantOps PASSED
      testDynamicQuantOps PASSED
      testQuantOps PASSED
      testNNUtilsOps PASSED
      testShuffleOps PASSED
      testVisionFunctionOps PASSED
      testLossFunctionOps PASSED
      testDistanceFunctionOps PASSED
      testSparseOps PASSED
      testDropoutOps PASSED
      testLinearOps PASSED
      testTransformerOps PASSED
      testRecurrentOps PASSED
      testNormalizationOps PASSED
      testActivationOps PASSED
      testPaddingOps PASSED
      testPoolingOps PASSED
      testConvolutionOps PASSED
      testTensorViewOps PASSED
      testTensorTypingOps PASSED
      testTensorIndexingOps PASSED
      testTensorCreationOps PASSED
      testTensorOps PASSED
      testSamplingOps PASSED
      testBlasLapackOps PASSED
      testSpectralOps PASSED
      testOtherMathOps PASSED
      testComparisonOps PASSED
      testReductionOps PASSED
      testPointwiseOps PASSED
      testMobileNetV2 PASSED
      testLiteInterpreter PASSED
      testModel PASSED
      testCoreML FAILED
        Saving FILE Automation_Output.txt (AUTOMATION_OUTPUT) at https://gha-artifacts.s3.amazonaws.com/device_farm/invalid-workflow-id/0/arn_aws_devicefarm_us-west-2_308535385114_artifact_b531574a-fb82-40ae-b687-8f0b81341ae0_fea59a70-c1ef-41ac-81c3-b8225b4aecf1_00003_00001_00036_00000_Automation_Output.txt
        Saving FILE Application_Output.txt (UNKNOWN) at https://gha-artifacts.s3.amazonaws.com/device_farm/invalid-workflow-id/0/arn_aws_devicefarm_us-west-2_308535385114_artifact_b531574a-fb82-40ae-b687-8f0b81341ae0_fea59a70-c1ef-41ac-81c3-b8225b4aecf1_00003_00001_00036_00001_Application_Output.txt
        Saving FILE Application_Crash_Report.ips (APPLICATION_CRASH_REPORT) at https://gha-artifacts.s3.amazonaws.com/device_farm/invalid-workflow-id/0/arn_aws_devicefarm_us-west-2_308535385114_artifact_b531574a-fb82-40ae-b687-8f0b81341ae0_fea59a70-c1ef-41ac-81c3-b8225b4aecf1_00003_00001_00036_00002_Application_Crash_Report.ips
        Saving FILE Video.mp4 (VIDEO) at https://gha-artifacts.s3.amazonaws.com/device_farm/invalid-workflow-id/0/arn_aws_devicefarm_us-west-2_308535385114_artifact_b531574a-fb82-40ae-b687-8f0b81341ae0_fea59a70-c1ef-41ac-81c3-b8225b4aecf1_00003_00001_00036_00003_Video.mp4
        Saving FILE Syslog.syslog (DEVICE_LOG) at https://gha-artifacts.s3.amazonaws.com/device_farm/invalid-workflow-id/0/arn_aws_devicefarm_us-west-2_308535385114_artifact_b531574a-fb82-40ae-b687-8f0b81341ae0_fea59a70-c1ef-41ac-81c3-b8225b4aecf1_00003_00001_00036_00004_Syslog.syslog
        Saving FILE TCP_dump_log.txt (RAW_FILE) at https://gha-artifacts.s3.amazonaws.com/device_farm/invalid-workflow-id/0/arn_aws_devicefarm_us-west-2_308535385114_artifact_b531574a-fb82-40ae-b687-8f0b81341ae0_fea59a70-c1ef-41ac-81c3-b8225b4aecf1_00003_00001_00036_00005_TCP_dump_log.txt
        Saving LOG ListArtifactType.log.json (MESSAGE_LOG) at https://gha-artifacts.s3.amazonaws.com/device_farm/invalid-workflow-id/0/arn_aws_devicefarm_us-west-2_308535385114_artifact_b531574a-fb82-40ae-b687-8f0b81341ae0_fea59a70-c1ef-41ac-81c3-b8225b4aecf1_00003_00001_00036_LOG_ListArtifactType.log.json
    Teardown Suite PASSED with stats {'total': 1, 'passed': 1, 'failed': 0, 'warned': 0, 'errored': 0, 'stopped': 0, 'skipped': 0}
  Apple iPhone 14 Pro PASSED
  Apple iPhone SE (2022) PASSED
```

.github/workflows/mobile_job.yml

@@ -0,0 +1,152 @@
+name: Run mobile tests on devices
+
+on:
+  workflow_call:
+    inputs:
+      job-name:
+        description: Name for the job, which is displayed in the GitHub UI
+        default: mobile-job
+        type: string
+      device-type:
+        description: The type of device (iOS or Android) to test against
+        type: string
+      runner:
+        description: The runner to run the test on
+        type: string
+      timeout:
+        description: Timeout for the job (in minutes)
+        required: false
+        default: 60
+        type: number
+      python-version:
+        description: Set the python version used in the job
+        required: false
+        type: string
+        default: '3.11'
+
+      # AWS Device Farm, this can be copied from AWS console and it's default to
+      # PyTorch project
+      project-arn:
+        description: The AWS Device Farm project where the test runs
+        default: 'arn:aws:devicefarm:us-west-2:308535385114:project:b531574a-fb82-40ae-b687-8f0b81341ae0'
+        type: string
+
+      # Pulling test-infra itself for device farm runner script
+      test-infra-repository:
+        description: Test infra repository to use
+        default: 'pytorch/test-infra'
+        type: string
+      test-infra-ref:
+        description: Test infra reference to use
+        default: ''
+        type: string
+
+      # iOS-specific inputs
+      ios-ipa-archive:
+        description: The name of the iOS app IPA archive to run the tests
+        required: false
+        type: string
+        default: ''
+      ios-xctest-zip:
+        description: The name of the iOS test suite itself
+        required: false
+        type: string
+        default: ''
+
+jobs:
+  job:
+    name: ${{ inputs.job-name }} (${{ inputs.device-type }})
+    runs-on: ${{ inputs.runner }}
+    timeout-minutes: ${{ inputs.timeout }}
+    permissions:
+      id-token: write
+      contents: read
+    steps:
+      - name: Clean workspace
+        run: |
+          echo "::group::Cleanup debug output"
+          rm -rfv "${GITHUB_WORKSPACE}"
+          mkdir -p "${GITHUB_WORKSPACE}"
+          echo "::endgroup::"
+
+      - name: Authenticate with AWS
+        uses: aws-actions/configure-aws-credentials@v3
+        with:
+          role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_mobile_job
+          aws-region: us-east-1
+
+      - name: Checkout repository (${{ inputs.test-infra-repository }}@${{ inputs.test-infra-ref }})
+        uses: actions/checkout@v3
+        with:
+          repository: ${{ inputs.test-infra-repository }}
+          ref: ${{ inputs.test-infra-ref }}
+          path: test-infra
+
+      - name: Setup miniconda
+        uses: ./test-infra/.github/actions/setup-miniconda
+        with:
+          python-version: ${{ inputs.python-version }}
+
+      - name: Install pip dependencies
+        shell: bash
+        working-directory: test-infra/tools/device-farm-runner
+        run: |
+          ${CONDA_RUN} pip install -r requirements.txt
+
+      - name: Download iOS app
+        if: ${{ inputs.device-type == 'ios' && inputs.ios-ipa-archive != '' }}
+        uses: actions/download-artifact@v3
+        with:
+          name: ${{ inputs.ios-ipa-archive }}
+          path: test-infra/tools/device-farm-runner/
+
+      - name: Download iOS test suite
+        if: ${{ inputs.device-type == 'ios' && inputs.ios-xctest-zip != '' }}
+        uses: actions/download-artifact@v3
+        with:
+          name: ${{ inputs.ios-xctest-zip }}
+          path: test-infra/tools/device-farm-runner/
+
+      - name: Verify iOS artifacts
+        if: ${{ inputs.device-type == 'ios' }}
+        shell: bash
+        working-directory: test-infra/tools/device-farm-runner
+        env:
+          IPA_ARCHIVE: ${{ inputs.ios-ipa-archive }}
+          XCTEST_ZIP: ${{ inputs.ios-xctest-zip }}
+        run: |
+          set -ex
+
+          if [ -z "${IPA_ARCHIVE}" ] || [ -z "${XCTEST_ZIP}" ]; then
+            echo "Missing IPA archive or xctest zip"
+            exit 1
+          fi
+
+          # NP: The suffix needs to be exact, otherwise, AWS will reject them
+          mv "${IPA_ARCHIVE}" ci.ipa
+          mv "${XCTEST_ZIP}" ci.xctest.zip
+
+          # Print the artifacts to manually verify them if needed
+          ls -lah ci.ipa ci.xctest.zip
+
+      - name: Run iOS tests on devices
+        if: ${{ inputs.device-type == 'ios' }}
+        shell: bash
+        working-directory: test-infra/tools/device-farm-runner
+        env:
+          PROJECT_ARN: ${{ inputs.project-arn }}
+          # For record keeping
+          JOB_NAME: ${{ inputs.job-name }}
+          DEVICE_TYPE: ${{ inputs.device-type }}
+          RUN_ID: ${{ github.run_id }}
+          RUN_ATTEMPT: ${{ github.run_attempt }}
+        run: |
+          set -ex
+
+          ${CONDA_RUN} python run_on_aws_devicefarm.py \
+            --project-arn "${PROJECT_ARN}" \
+            --app-file ci.ipa \
+            --xctest-file ci.xctest.zip \
+            --name-prefix "${JOB_NAME}-${DEVICE_TYPE}" \
+            --workflow-id "${RUN_ID}" \
+            --workflow-attempt "${RUN_ATTEMPT}"

.github/workflows/test_mobile_job.yml

@@ -0,0 +1,50 @@
+name: Test mobile workflow
+
+on:
+  pull_request:
+    paths:
+      - .github/workflows/mobile_job.yml
+      - .github/workflows/test_mobile_job.yml
+      - tools/device-farm-runner/**
+  workflow_dispatch:
+
+jobs:
+  setup-ios-job:
+    runs-on: ubuntu-latest
+    env:
+      IPA_ARCHIVE: TestApp.ipa
+      XCTEST_ZIP: TestAppTests.xctest.zip
+    steps:
+      - name: Download the prebuilt PyTorch iOS app and test suite from S3
+        run: |
+          set -ex
+          wget -q "https://ossci-assets.s3.amazonaws.com/${IPA_ARCHIVE}"
+          wget -q "https://ossci-assets.s3.amazonaws.com/${XCTEST_ZIP}"
+          # Print the artifacts to manually verify them if needed
+          ls -lah "${IPA_ARCHIVE}" "${XCTEST_ZIP}"
+
+      - name: Upload the iOS app
+        uses: actions/upload-artifact@v3
+        with:
+          name: ${{ env.IPA_ARCHIVE }}
+          path: ${{ env.IPA_ARCHIVE }}
+
+      - name: Upload the test suite
+        uses: actions/upload-artifact@v3
+        with:
+          name: ${{ env.XCTEST_ZIP }}
+          path: ${{ env.XCTEST_ZIP }}
+
+  test-ios-job:
+    needs: setup-ios-job
+    permissions:
+      id-token: write
+      contents: read
+    uses: ./.github/workflows/mobile_job.yml
+    with:
+      device-type: ios
+      # For iOS testing, the runner just needs to call AWS Device Farm, so there is no need to run this on macOS
+      runner: ubuntu-latest
+      project-arn: arn:aws:devicefarm:us-west-2:308535385114:project:b531574a-fb82-40ae-b687-8f0b81341ae0
+      ios-ipa-archive: TestApp.ipa
+      ios-xctest-zip: TestAppTests.xctest.zip

tools/device-farm-runner/requirements.txt

@@ -0,0 +1,3 @@
+awscli==1.32.50
+boto3==1.34.50
+requests==2.28.1