Create runtime clone to manually kick off full test runs #61641

steveisok · 2021-11-16T00:22:25Z

This change allows devs to manually kick off full test runs on the configurations that only execute smoke tests per PR.

/azp runtime-manual will do the trick

[skip ci]

ghost · 2021-11-16T00:22:31Z

Tagging subscribers to this area: @Anipik, @safern, @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Issue Details

null

Author:	steveisok
Assignees:	-
Labels:	`area-Infrastructure-libraries`
Milestone:	-

…nually

jkoritzinsky

This new pipeline has no differences from the runtime pipeline (at least in the YAML). What's the use case?

If this is the new "full" pipeline, then I would highly prefer if we were to manually pre-evaluate the 'isFullMatrix' variable and make the runtime-manual pipeline have the jobs from when it is true and the runtime pipeline have the jobs from when it is false. Otherwise we're adding complexity on complexity and making our build systems more difficult to understand instead of reducing complexity.

safern · 2021-11-16T00:44:24Z

eng/pipelines/runtime-template.yml

+          eq(dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'], true),
+          eq(dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'], true),
+          eq(dependencies.evaluate_paths.outputs['SetPathVars_installer.containsChange'], true),
+          eq(variables['isManualOrIsNotPR'], true),


These are already run as part of PRs; I guess the difference is that we would run the full tests not only the smoke tests?

@jkoritzinsky this is the main difference. It will only run the mono jobs that were turned into smoke tests like this one, but with full test mode.

Yes, that is correct.

I would highly prefer if this pipeline could just include these jobs instead of adding another variable into the template.

We were already at the point where it's extremely difficult to understand what is triggered where. For example, this new pipeline would include the following job as the equivalent for the job this comment is attached to:

# # Build for Browser/wasm, with EnableAggressiveTrimming=true # - template: /eng/pipelines/common/platform-matrix.yml parameters: jobTemplate: /eng/pipelines/common/global-build-job.yml helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml buildConfig: Release runtimeFlavor: mono platforms: - Browser_wasm jobParameters: testGroup: innerloop nameSuffix: AllSubsets_Mono_EAT buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:RunSmokeTestsOnly=false /p:EnableAggressiveTrimming=true /p:BuildAOTTestsOnHelix=true /p:RunAOTCompilation=false timeoutInMinutes: 180 # extra steps, run tests extraStepsTemplate: /eng/pipelines/libraries/helix.yml extraStepsParameters: creator: dotnet-bot testRunNamePrefixSuffix: Mono_$(_BuildConfig) extraHelixArguments: /p:NeedsToBuildWasmAppsOnHelix=true /p:RunSmokeTestsOnly=false scenarios: - normal

steveisok · 2021-11-16T00:58:41Z

This new pipeline has no differences from the runtime pipeline (at least in the YAML). What's the use case?

If this is the new "full" pipeline, then I would highly prefer if we were to manually pre-evaluate the 'isFullMatrix' variable and make the runtime-manual pipeline have the jobs from when it is true and the runtime pipeline have the jobs from when it is false. Otherwise we're adding complexity on complexity and making our build systems more difficult to understand instead of reducing complexity.

The intent is to allow devs to manually trigger full test runs for all legs. The reason why this is important is that some configurations only run System.Runtime tests on PR's. If a dev wants to add a test to wasm, for example, there is no good way until the rolling build to actually get a pass / fail indication.

I expect to have a follow up PR to make this a bit more streamlined. Merging this PR will give us something usable while we work out the best way.

jkoritzinsky · 2021-11-16T01:19:48Z

The intent is to allow devs to manually trigger full test runs for all legs. The reason why this is important is that some configurations only run System.Runtime tests on PR's. If a dev wants to add a test to wasm, for example, there is no good way until the rolling build to actually get a pass / fail indication.

If that is the use case, then this pipeline should explicitly define only the exact jobs that it runs. If it needs to run only a subset of the jobs, we should include only those jobs in the pipeline YAML. We should not just keep adding new variables to define when we run or skip different subsets of jobs, especially for manually triggered pipelines.

And following on the conversation we had that led to #61588, we shouldn't keep perpetuating the "some jobs in PR, some in CI" design as it adds complexity and makes knowing how to run a given job in PR vs CI difficult. We should instead have these jobs for these platforms on a separate pipeline that is CI or manual/comment trigger only.

I expect to have a follow up PR to make this a bit more streamlined. Merging this PR will give us something usable while we work out the best way.

I fear that this is one of those scenarios where "later" will continue to be pushed back due to more important work and our CI infrastructure will continue to become increasingly inscrutable to anyone who is not deeply involved in it regularly.

steveisok · 2021-11-16T02:55:46Z

If that is the use case, then this pipeline should explicitly define only the exact jobs that it runs. If it needs to run only a subset of the jobs, we should include only those jobs in the pipeline YAML. We should not just keep adding new variables to define when we run or skip different subsets of jobs, especially for manually triggered pipelines.

I'm after whatever will be an acceptable interim solution. If duplicating parts of runtime.yml will get us there, fine by me.

I fear that this is one of those scenarios where "later" will continue to be pushed back due to more important work and our CI infrastructure will continue to become increasingly inscrutable to anyone who is not deeply involved in it regularly.

With the mono configurations mostly under smoke tests, I'm not interested in punting on the right solution. I'd like to get it right.

jkoritzinsky

@steveisok I think the work to get a working build without adding complexity is not expensive enough to warrant the current proposed interim solution. The code block below should include all of the jobs from the shared pipeline conditioned on isManualOrIsNotPR. As a result, the following snippet should define all of the jobs needed in the runtime-manual.yml pipeline if the current pipeline definitions are accurate.

variables:
  - template: /eng/pipelines/common/variables.yml

jobs:
#
# Build for Browser/wasm, with EnableAggressiveTrimming=true
#
- template: /eng/pipelines/common/platform-matrix.yml
  parameters:
    jobTemplate: /eng/pipelines/common/global-build-job.yml
    helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml
    buildConfig: Release
    runtimeFlavor: mono
    platforms:
    - Browser_wasm
    variables:
      # map dependencies variables to local variables
      - name: librariesContainsChange
        value: $[ dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'] ]
      - name: monoContainsChange
        value: $[ dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'] ]
    jobParameters:
      testGroup: innerloop
      nameSuffix: AllSubsets_Mono_EAT
      buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:RunSmokeTestsOnly=false /p:EnableAggressiveTrimming=true /p:BuildAOTTestsOnHelix=true /p:RunAOTCompilation=false
      timeoutInMinutes: 180
      # extra steps, run tests
      extraStepsTemplate: /eng/pipelines/libraries/helix.yml
      extraStepsParameters:
        creator: dotnet-bot
        testRunNamePrefixSuffix: Mono_$(_BuildConfig)
        extraHelixArguments: /p:NeedsToBuildWasmAppsOnHelix=true /p:RunSmokeTestsOnly=false
        scenarios:
        - normal

#
# Build for Browser/wasm with RunAOTCompilation=true
#
- template: /eng/pipelines/common/platform-matrix.yml
  parameters:
    jobTemplate: /eng/pipelines/common/global-build-job.yml
    helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml
    buildConfig: Release
    runtimeFlavor: mono
    platforms:
    - Browser_wasm
    variables:
      # map dependencies variables to local variables
      - name: librariesContainsChange
        value: $[ dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'] ]
      - name: monoContainsChange
        value: $[ dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'] ]
    jobParameters:
      testGroup: innerloop
      nameSuffix: AllSubsets_Mono_AOT
      buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:RunSmokeTestsOnly=false /p:EnableAggressiveTrimming=true /p:BuildAOTTestsOnHelix=true /p:RunAOTCompilation=true
      timeoutInMinutes: 180
      # extra steps, run tests
      extraStepsTemplate: /eng/pipelines/libraries/helix.yml
      extraStepsParameters:
        creator: dotnet-bot
        testRunNamePrefixSuffix: Mono_$(_BuildConfig)
        extraHelixArguments: /p:NeedsToBuildWasmAppsOnHelix=true /p:RunSmokeTestsOnly=false
        scenarios:
        - normal

#
# Build the whole product using Mono and run runtime tests
#
- template: /eng/pipelines/common/platform-matrix.yml
  parameters:
    jobTemplate: /eng/pipelines/common/global-build-job.yml
    helixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.yml
    buildConfig: Release
    runtimeFlavor: mono
    platforms:
    - Browser_wasm
    variables:
      - ${{ if and(eq(variables['System.TeamProject'], 'public'), eq(variables['Build.Reason'], 'PullRequest')) }}:
        - name: _HelixSource
          value: pr/dotnet/runtime/$(Build.SourceBranch)
      - ${{ if and(eq(variables['System.TeamProject'], 'public'), ne(variables['Build.Reason'], 'PullRequest')) }}:
        - name: _HelixSource
          value: ci/dotnet/runtime/$(Build.SourceBranch)
      - name: timeoutPerTestInMinutes
        value: 10
      - name: timeoutPerTestCollectionInMinutes
        value: 200
    jobParameters:
      testGroup: innerloop
      nameSuffix: AllSubsets_Mono_RuntimeTests
      buildArgs: -s mono+libs -c $(_BuildConfig)
      timeoutInMinutes: 180
      # NOTE: Per PR test execution is not recommended for mobile runtime tests
      # extra steps, run tests
      extraStepsTemplate: /eng/pipelines/common/templates/runtimes/wasm-runtime-and-send-to-helix.yml
      extraStepsParameters:
        creator: dotnet-bot
        testRunNamePrefixSuffix: Mono_$(_BuildConfig)

#
# Build the whole product using Mono for Android and run runtime tests with Android emulator
#
- template: /eng/pipelines/common/platform-matrix.yml
  parameters:
    jobTemplate: /eng/pipelines/common/global-build-job.yml
    helixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.yml
    buildConfig: Release
    runtimeFlavor: mono
    platforms:
    - Android_x64
    variables:
      - ${{ if and(eq(variables['System.TeamProject'], 'public'), eq(variables['Build.Reason'], 'PullRequest')) }}:
        - name: _HelixSource
          value: pr/dotnet/runtime/$(Build.SourceBranch)
      - ${{ if and(eq(variables['System.TeamProject'], 'public'), ne(variables['Build.Reason'], 'PullRequest')) }}:
        - name: _HelixSource
          value: ci/dotnet/runtime/$(Build.SourceBranch)
      - name: timeoutPerTestInMinutes
        value: 60
      - name: timeoutPerTestCollectionInMinutes
        value: 180
    jobParameters:
      testGroup: innerloop
      nameSuffix: AllSubsets_Mono_RuntimeTests
      buildArgs: -s mono+libs -c $(_BuildConfig)
      timeoutInMinutes: 240
      # NOTE: Per PR test execution is not recommended for mobile runtime tests
      # extra steps, run tests
      extraStepsTemplate: /eng/pipelines/common/templates/runtimes/android-runtime-and-send-to-helix.yml
      extraStepsParameters:
        creator: dotnet-bot
        testRunNamePrefixSuffix: Mono_$(_BuildConfig)

#
# Mono Test builds with CoreCLR runtime tests using live libraries debug build
- template: /eng/pipelines/common/platform-matrix.yml
  parameters:
    jobTemplate: /eng/pipelines/common/templates/runtimes/build-test-job.yml
    buildConfig: release
    runtimeFlavor: mono
    platforms:
    - CoreClrTestBuildHost # Either OSX_x64 or Linux_x64
    jobParameters:
      testGroup: innerloop
      liveLibrariesBuildConfig: ${{ variables.debugOnPrReleaseOnRolling }}
      liveRuntimeBuildConfig: release

#
# Mono CoreCLR runtime Test executions using live libraries in jit mode
- template: /eng/pipelines/common/platform-matrix.yml
  parameters:
    jobTemplate: /eng/pipelines/common/templates/runtimes/run-test-job.yml
    buildConfig: release
    runtimeFlavor: mono
    platforms:
    - OSX_x64
    - Linux_arm64
    helixQueueGroup: pr
    helixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.yml
    jobParameters:
      testGroup: innerloop
      liveLibrariesBuildConfig: ${{ variables.debugOnPrReleaseOnRolling }}
      liveRuntimeBuildConfig: release
      runtimeVariant: minijit

#
# Mono CoreCLR runtime Test executions using live libraries in interpreter mode
- template: /eng/pipelines/common/platform-matrix.yml
  parameters:
    jobTemplate: /eng/pipelines/common/templates/runtimes/run-test-job.yml
    buildConfig: release
    runtimeFlavor: mono
    platforms:
    - OSX_x64
    - Linux_arm64
    helixQueueGroup: pr
    helixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.yml
    jobParameters:
      testGroup: innerloop
      liveLibrariesBuildConfig: ${{ variables.debugOnPrReleaseOnRolling }}
      liveRuntimeBuildConfig: release
      runtimeVariant: monointerpreter

#
# Mono CoreCLR runtime Test executions using live libraries and LLVM AOT
#
- template: /eng/pipelines/common/platform-matrix.yml
  parameters:
    jobTemplate: /eng/pipelines/common/templates/runtimes/run-test-job.yml
    buildConfig: release
    runtimeFlavor: mono
    platforms:
    - Linux_x64
    # Disabled pending outcome of https://github.com/dotnet/runtime/issues/60234 investigation
    #- Linux_arm64
    helixQueueGroup: pr
    helixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.yml
    jobParameters:
      testGroup: innerloop
      liveLibrariesBuildConfig: ${{ variables.debugOnPrReleaseOnRolling }}
      liveRuntimeBuildConfig: release
      runtimeVariant: llvmaot

#
# Mono CoreCLR runtime Test executions using live libraries and LLVM Full AOT
#
- template: /eng/pipelines/common/platform-matrix.yml
  parameters:
    jobTemplate: /eng/pipelines/common/templates/runtimes/run-test-job.yml
    buildConfig: release
    runtimeFlavor: mono
    platforms:
    - Linux_x64
    # Disabled pending outcome of https://github.com/dotnet/runtime/issues/60234 investigation
    #- Linux_arm64
    helixQueueGroup: pr
    helixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.yml
    jobParameters:
      testGroup: innerloop
      liveLibrariesBuildConfig: ${{ variables.debugOnPrReleaseOnRolling }}
      liveRuntimeBuildConfig: release
      runtimeVariant: llvmfullaot

Please change the runtime-manual pipeline to use this snippet (+ any changes required to get it to work) and change the runtime.yml pipeline back to the way it is on main.

jkoritzinsky · 2021-11-16T20:50:47Z

Let's make sure to run this pipeline at least once before merging so we know it works.

steveisok · 2021-11-16T20:51:54Z

/azp run runtime-manual

azure-pipelines · 2021-11-16T20:52:09Z

Azure Pipelines failed to run 1 pipeline(s).

steveisok · 2021-11-16T20:53:44Z

Let's make sure to run this pipeline at least once before merging so we know it works.

I wondered if there were any runtime test deps and it looks like there are. They may be acceptable to exclude for the time being.

steveisok · 2021-11-17T15:14:41Z

/azp run runtime-manual

azure-pipelines · 2021-11-17T15:14:55Z

Azure Pipelines successfully started running 1 pipeline(s).

lewing

👏

safern · 2021-11-17T17:40:03Z

Wild idea, if we are using a separate template, why not include these as part of staging-manual? That way mono devs an people that care about this don't have to trigger two separate pipelines?

steveisok · 2021-11-17T17:52:21Z

Wild idea, if we are using a separate template, why not include these as part of staging-manual? That way mono devs an people that care about this don't have to trigger two separate pipelines?

I'd rather call it runtime-manual and fold runtime-staging-manual into it. With the way the PR builds have changed, there are less reasons to have builds in runtime-staging.

Good idea :-)

safern · 2021-11-17T17:54:48Z

I'd rather call it runtime-manual and fold runtime-staging-manual into it. With the way the PR builds have changed, there are less reasons to have builds in runtime-staging.

Like that idea.

steveisok · 2021-11-17T19:47:19Z

/azp run runtime-manual

azure-pipelines · 2021-11-17T19:47:34Z

Azure Pipelines successfully started running 1 pipeline(s).

…time-pipeline

Create runtime clone to manually kick off full test runs

133562c

[skip ci]

dotnet-issue-labeler bot added the area-Infrastructure-libraries label Nov 16, 2021

steveisok requested review from akoeplinger, safern, lewing and radical November 16, 2021 00:34

Add runtime template and allow smoke test configurations to be run ma…

4dc44d0

…nually

steveisok force-pushed the stpfiste/manual-runtime-pipeline branch from 378e3ec to 4dc44d0 Compare November 16, 2021 00:42

jkoritzinsky reviewed Nov 16, 2021

View reviewed changes

safern reviewed Nov 16, 2021

View reviewed changes

jkoritzinsky requested changes Nov 16, 2021

View reviewed changes

Revert runtime template and copy smoke test jobs into runtime-manual

be752a2

jkoritzinsky approved these changes Nov 16, 2021

View reviewed changes

safern approved these changes Nov 16, 2021

View reviewed changes

Exclude desktop runtime tests

3312c18

steveisok force-pushed the stpfiste/manual-runtime-pipeline branch from 1f18313 to 3312c18 Compare November 16, 2021 23:56

akoeplinger approved these changes Nov 17, 2021

View reviewed changes

lewing approved these changes Nov 17, 2021

View reviewed changes

Fold runtime-staging-manual into runtime-manual

794a278

Steve Pfister added 2 commits November 17, 2021 16:17

Merge remote-tracking branch 'upstream/main' into stpfiste/manual-run…

c4a87dd

…time-pipeline

Skip Android regex test failure

49e2ad5

steveisok merged commit 95e3144 into main Nov 18, 2021

steveisok deleted the stpfiste/manual-runtime-pipeline branch November 18, 2021 02:51

steveisok mentioned this pull request Nov 18, 2021

[mono] Create separate yml that can trigger full test runs on demand #61132

Closed

AndyAyersMS mentioned this pull request Nov 22, 2021

arm64 jitstress failing with encoding_found assert #61944

Closed

ghost locked as resolved and limited conversation to collaborators Dec 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create runtime clone to manually kick off full test runs #61641

Create runtime clone to manually kick off full test runs #61641

steveisok commented Nov 16, 2021 •

edited

Loading

ghost commented Nov 16, 2021

jkoritzinsky left a comment

safern Nov 16, 2021

safern Nov 16, 2021

steveisok Nov 16, 2021

jkoritzinsky Nov 16, 2021

steveisok commented Nov 16, 2021

jkoritzinsky commented Nov 16, 2021

steveisok commented Nov 16, 2021

jkoritzinsky left a comment

jkoritzinsky commented Nov 16, 2021

steveisok commented Nov 16, 2021

azure-pipelines bot commented Nov 16, 2021

steveisok commented Nov 16, 2021

steveisok commented Nov 17, 2021

azure-pipelines bot commented Nov 17, 2021

lewing left a comment

safern commented Nov 17, 2021

steveisok commented Nov 17, 2021 •

edited

Loading

safern commented Nov 17, 2021

steveisok commented Nov 17, 2021

azure-pipelines bot commented Nov 17, 2021

Create runtime clone to manually kick off full test runs #61641

Create runtime clone to manually kick off full test runs #61641

Conversation

steveisok commented Nov 16, 2021 • edited Loading

ghost commented Nov 16, 2021

jkoritzinsky left a comment

Choose a reason for hiding this comment

safern Nov 16, 2021

Choose a reason for hiding this comment

safern Nov 16, 2021

Choose a reason for hiding this comment

steveisok Nov 16, 2021

Choose a reason for hiding this comment

jkoritzinsky Nov 16, 2021

Choose a reason for hiding this comment

steveisok commented Nov 16, 2021

jkoritzinsky commented Nov 16, 2021

steveisok commented Nov 16, 2021

jkoritzinsky left a comment

Choose a reason for hiding this comment

jkoritzinsky commented Nov 16, 2021

steveisok commented Nov 16, 2021

azure-pipelines bot commented Nov 16, 2021

steveisok commented Nov 16, 2021

steveisok commented Nov 17, 2021

azure-pipelines bot commented Nov 17, 2021

lewing left a comment

Choose a reason for hiding this comment

safern commented Nov 17, 2021

steveisok commented Nov 17, 2021 • edited Loading

safern commented Nov 17, 2021

steveisok commented Nov 17, 2021

azure-pipelines bot commented Nov 17, 2021

steveisok commented Nov 16, 2021 •

edited

Loading

steveisok commented Nov 17, 2021 •

edited

Loading