VAULT-34822: Add pipeline github list changed-files (#30100)

* VAULT-34822: Add `pipeline github list changed-files`

Add a new `github list changed-files` sub-command to `pipeline` command and
integrate it into the pipeline. This replaces our previous
`changed-files.sh` script.

This command works quite a bit differently than the full checkout and
diff based solution we used before. Instead of checking out the base ref
and head ref and comparing a diff, we now provide either a pull request
number or git commit SHA and use the Github REST API to determine the
changed files.

This approach has several benefits:
  - Not requiring a local checkout of the repo to get the list of
    changed files. This yields a significant perfomance improvement in
    `setup` jobs where we typically determine the changed files list.
  - The CLI supports both PRs and commit SHAs.
  - The implementation is portable and doesn't require any system tools
    like `git` or `bash` to be installed.
  - A much more advanced system for adding group metadata to the changed
    files. These groupings are going to be used heavily in future
    pipeline automation work and will be used to make required jobs
    smarter.

The theoretical drawbacks:
   - It requires a GITHUB_TOKEN and only works for remote branches or
     commits in Github. We could eventually add a local diff sub-command
     or option to work locally, but that was not required for what we're
     trying to achieve here.

While the groupings that I added in this change are quite rudimentary,
the system will allow us to add additional groups with very little
overhead. I tried to make this change more or less a port of the old
system to enable future work. I did include one small change of
behavior, which is that we now build all extended targets if the
`go.mod` or `go.sum` files change. We do this to ensure that dependency
changes don't subtly result in some extended platform breakage.

Signed-off-by: Ryan Cragun <[email protected]>

Author: ryancragun

.github/actions/changed-files/action.yml

@@ -2,75 +2,45 @@
 # SPDX-License-Identifier: BUSL-1.1
 
 ---
-name: Determine what files changed between two git referecnes.
+name: Determine what files have changed on either a pull request or commit.
 description: |
-  Determine what files have changed between two git references. If the github.event_type is
-  pull_request we'll compare the github.base_ref (merge target) and pull request head SHA.
-  For other event types we'll gather the changed files from the most recent commit. This allows
-  us to support PR and merge workflows.
+  Determine what files have changed on either a pull request or commit.
+  Writes the list of files to
+
+inputs:
+  github-token:
+    description: A preferred Github token to access private modules if necessary.
 
 outputs:
-  app-changed:
-    description: Whether or not the vault Go app was modified.
-    value: ${{ steps.changed-files.outputs.app-changed }}
-  docs-changed:
-    description: Whether or not the documentation was modified.
-    value: ${{ steps.changed-files.outputs.docs-changed }}
-  ui-changed:
-    description: Whether or not the web UI was modified.
-    value: ${{ steps.changed-files.outputs.ui-changed }}
-  autopilot-changed:
-    description: Whether or not files pertaining to Autopilot were modified.
-    value: ${{ steps.changed-files.outputs.autopilot-changed }}
-  files:
-    description: All of the file names that changed.
-    value: ${{ steps.changed-files.outputs.files }}
+  changed-files:
+    description: All of the files that changed.
+    value: ${{ steps.changed-files.outputs.changed-files }}
 
 runs:
   using: composite
   steps:
-    - id: ref
+    - id: changed-files-set-up-pipeline
+      name: Set up the pipeline tool
+      uses: ./.github/actions/set-up-pipeline
+      with:
+        github-token: ${{ inputs.github-token || github.token }}
+    - id: changed-files
+      name: Determine the changed files
       shell: bash
-      name: ref
+      env:
+        GITHUB_TOKEN: ${{ inputs.github-token || github.token }}
       run: |
-        # Determine our desired checkout ref.
-        #
-        # * If the trigger event is pull_request we will default to a magical merge SHA that Github
-        #   creates. This SHA is the product of what merging our PR into the merge target branch at
-        #   at the point in time when we created the PR. When you push a change to a PR branch
-        #   Github updates this branch if it can. When you rebase a PR it updates this branch.
-        #
-        # * If the trigger event is pull_request and a `checkout-head` tag is present or the
-        #   checkout-head input is set, we'll use HEAD of the PR branch instead of the magical
-        #   merge SHA.
-        #
-        # * If the trigger event is a push (merge) then we'll get the latest commit that was pushed.
-        #
-        # * For anything any other event type we'll default to whatever is default in Github.
+        # Get a list of changed files and write the "changed-files" output to $GITHUB_OUTPUT
         if [ '${{ github.event_name }}' = 'pull_request' ]; then
-          checkout_ref='${{ github.event.pull_request.head.sha }}'
-        elif [ '${{ github.event_name }}' = 'push' ]; then
-          # Our checkout ref for any other event type should default to the github ref.
-          checkout_ref='${{ github.event.after && github.event.after || github.event.push.after }}'
+          pipeline github list changed-files \
+            --owner hashicorp \
+            --repo '${{ github.event.pull_request.head.repo.name }}' \
+            --pr '${{ github.event.pull_request.number }}' \
+            --github-output
         else
-          checkout_ref='${{ github.ref }}'
+          pipeline github list changed-files \
+            --owner hashicorp \
+            --repo '${{ github.event.repository.name }}' \
+            --commit '${{ github.event.after && github.event.after || github.event.push.after && github.event.push.after || github.sha }}' \
+            --github-output
         fi
-        echo "ref=${checkout_ref}" | tee -a "$GITHUB_OUTPUT"
-    - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
-      with:
-        repository: ${{ github.repository }}
-        path: "changed-files"
-        # The fetch-depth could probably be optimized at some point. It's currently set to zero to
-        # ensure that we have a successfull diff, regardless of how many commits might be present
-        # present between the two references we're comparing. It would be nice to change this
-        # depending on the number of commits by using the push.commits and/or pull_request.commits
-        # payload fields, however, they have different behavior and limitations. For now we'll do
-        # the slow but sure thing of getting the whole repository.
-        fetch-depth: 0
-        ref: ${{ steps.ref.outputs.ref }}
-    - id: changed-files
-      name: changed-files
-      # This script writes output values to $GITHUB_OUTPUT and STDOUT
-      shell: bash
-      run: ./.github/scripts/changed-files.sh ${{ github.event_name }} ${{ github.ref_name }} ${{ github.base_ref }}
-      working-directory: changed-files

.github/actions/set-up-pipeline/action.yml

@@ -14,7 +14,7 @@ runs:
   steps:
     - uses: ./.github/actions/set-up-go
       with:
-        github-token: ${{ inputs.github-token }}
+        github-token: ${{ inputs.github-token || github.token }}
         no-restore: true # Don't download vault's modules for pipeline
     - name: pipeline-metadata
       id: pipeline-metadata

.github/scripts/changed-files.sh

@@ -83,33 +83,32 @@ jobs:
       github.event_name == 'schedule' ||
       (github.event_name == 'pull_request' && github.event.pull_request.draft == false)
     runs-on: ${{ github.repository == 'hashicorp/vault' && 'ubuntu-latest' || fromJSON('["self-hosted","linux","small"]') }}
+    permissions:
+      id-token: write # vault-auth
+      contents: read
     outputs:
-      app-changed: ${{ steps.changed-files.outputs.app-changed }}
       build-date: ${{ steps.metadata.outputs.vault-build-date }}
+      changed-files: ${{ steps.changed-files.outputs.changed-files }}
       checkout-ref: ${{ steps.checkout.outputs.ref }}
       compute-build: ${{ steps.metadata.outputs.compute-build }}
       compute-build-compat: ${{ steps.metadata.outputs.compute-build-compat }}
       compute-build-ui: ${{ steps.metadata.outputs.compute-build-ui }}
       compute-small: ${{ steps.metadata.outputs.compute-small }}
-      docs-changed: ${{ steps.changed-files.outputs.docs-changed }}
       is-draft: ${{ steps.metadata.outputs.is-draft }}
       is-enterprise: ${{ steps.metadata.outputs.is-enterprise }}
       is-fork: ${{ steps.metadata.outputs.is-fork }}
       labels: ${{ steps.metadata.outputs.labels }}
-      ui-changed: ${{ steps.changed-files.outputs.ui-changed }}
       vault-binary-name: ${{ steps.metadata.outputs.vault-binary-name }}
       vault-revision: ${{ steps.metadata.outputs.vault-revision }}
       vault-version: ${{ steps.metadata.outputs.vault-version }}
       vault-version-metadata: ${{ steps.metadata.outputs.vault-version-metadata }}
       vault-version-package: ${{ steps.metadata.outputs.vault-version-package }}
       workflow-trigger: ${{ steps.metadata.outputs.workflow-trigger }}
     steps:
-      # Run the changed-files action to determine what Git reference we should check out
       - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
-      - uses: ./.github/actions/changed-files
-        id: changed-files
+      # Make sure we check out correct ref based on PR labels and such
       - uses: ./.github/actions/checkout
-        id: checkout # Make sure we check out correct ref after checking changed files
+        id: checkout
       # Get the vault version metadata
       - uses: hashicorp/actions-set-product-version@v2
         id: set-product-version
@@ -120,9 +119,29 @@ jobs:
         id: metadata
         with:
           vault-version: ${{ steps.set-product-version.outputs.product-version }}
+      # Get the elevated github token
+      - if: steps.metadata.outputs.is-enterprise == 'true'
+        id: vault-auth
+        name: Vault Authenticate
+        run: vault-auth
+      - if: steps.metadata.outputs.is-enterprise == 'true'
+        id: vault-secrets
+        name: Fetch Vault Secrets
+        uses: hashicorp/vault-action@d1720f055e0635fd932a1d2a48f87a666a57906c # v3.0.0
+        with:
+          url: ${{ steps.vault-auth.outputs.addr }}
+          caCertificate: ${{ steps.vault-auth.outputs.ca_certificate }}
+          token: ${{ steps.vault-auth.outputs.token }}
+          secrets: |
+            kv/data/github/${{ github.repository }}/github-token token | ELEVATED_GITHUB_TOKEN;
+      # Determine the changed files
+      - uses: ./.github/actions/changed-files
+        id: changed-files
+        with:
+          github-token: ${{ steps.metadata.outputs.is-enterprise != 'true' && secrets.ELEVATED_GITHUB_TOKEN || steps.vault-secrets.outputs.ELEVATED_GITHUB_TOKEN }}
+      # Make sure all required Go modules are cached at this point. We don't want all of the Go
+      # tests and build jobs to download modules and race to upload them to the cache.
       - uses: ./.github/actions/set-up-go
-        # Make sure all required Go modules are cached at this point. We don't want all of the Go
-        # tests and build jobs to download modules and race to upload them to the cache.
         name: Ensure Go modules are cached
         with:
           github-token: ${{ secrets.ELEVATED_GITHUB_TOKEN }}
@@ -150,8 +169,8 @@ jobs:
         needs.setup.outputs.workflow-trigger == 'pull_request' &&
         needs.setup.outputs.is-draft == 'false' &&
         (
-          needs.setup.outputs.ui-changed == 'true' ||
-          needs.setup.outputs.app-changed == 'true'
+          contains(fromJSON(needs.setup.outputs.changed-files).groups, 'ui') ||
+          contains(fromJSON(needs.setup.outputs.changed-files).groups, 'app')
         )
       )
     needs: setup
@@ -200,7 +219,7 @@ jobs:
     #   * The build/all label is present on a pull request or push.
     if: |
       needs.setup.outputs.workflow-trigger == 'schedule' ||
-      needs.setup.outputs.app-changed == 'true' ||
+      contains(fromJSON(needs.setup.outputs.changed-files).groups, 'app') ||
       contains(fromJSON(needs.setup.outputs.labels), 'build/all')
     needs:
       - setup
@@ -210,7 +229,7 @@ jobs:
     with:
       # The inputs defined here must be supported in both the build-artifacts-ce and
       # build-artifacts-ent workflows. The implementations should seek to keep a compatible interface.
-      build-all: ${{ contains(fromJSON(needs.setup.outputs.labels), 'build/all') || needs.setup.outputs.workflow-trigger == 'schedule' }}
+      build-all: ${{ contains(fromJSON(needs.setup.outputs.labels), 'build/all') || needs.setup.outputs.workflow-trigger == 'schedule' || contains(fromJSON(needs.setup.outputs.changed-files).groups, 'gomod') }}
       build-date: ${{ needs.setup.outputs.build-date }}
       checkout-ref: ${{ needs.setup.outputs.checkout-ref }}
       compute-build: ${{ needs.setup.outputs.compute-build }}