diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 18ab7650..a57a31c0 100755 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -1,68 +1,161 @@ # Carpentries Workflows -This directory contains workflows to be used for Lessons using the {sandpaper} -lesson infrastructure. Two of these workflows require R (`sandpaper-main.yaml` -and `pr-receive.yaml`) and the rest are bots to handle pull request management. +This directory contains workflows to be used for Lessons using the Carpentries Workbench lesson infrastructure. -These workflows will likely change as {sandpaper} evolves, so it is important to -keep them up-to-date. To do this in your lesson you can do the following in your -R console: +The three `docker-` workflows build lessons and maintain packages. +The workflows run using the [workbench-docker](https://github.com/carpentries/workbench-docker) container. +This container comprises prebuilt and installed dependencies of the core Workbench packages, i.e. sandpaper, pegboard and varnish. + +Two `update-` workflows handle: + - checking for new renv packages and creating a Pull Request (PR) when a renv.lock file updates (`update-cache.yaml`) + - checking for updated versions of these workflow files (`update-workflows.yaml`) + +The rest of the `pr-` workflows handle pull request management via base GitHub Actions. + +For Carpentries Core Curriculum lessons across our lesson programmes, maintenance of these workflows should be minimal. +For your own lesson repositories, it is important to understand the different workflows and what they do. + +## Managing Updates + +By using prebuilt Docker containers that are managed by the Carpentries core Workbench maintainers, these workflows are designed to be rarely updated. + +However, is important to be able to keep them up-to-date when appropriate. +You can do this locally using your own R and Workbench installation, or via the "04 Maintain: Update Workflow Files" (`update-workflows.yaml`) GitHub Action. + +### Updating locally + +In a terminal/git bash, navigate to the lesson folder where you want to update the workflows. + +Then, start an R session and: ```r # Install/Update sandpaper -options(repos = c(carpentries = "https://carpentries.r-universe.dev/", - CRAN = "https://cloud.r-project.org")) +options(repos = c(carpentries = "https://carpentries.r-universe.dev/", CRAN = "https://cloud.r-project.org")) install.packages("sandpaper") # update the workflows in your lesson library("sandpaper") -update_github_workflows() +sandpaper::update_github_workflows() +quit() ``` -Inside this folder, you will find a file called `sandpaper-version.txt`, which -will contain a version number for sandpaper. This will be used in the future to -alert you if a workflow update is needed. +And then in a bash prompt/git bash terminal: -What follows are the descriptions of the workflow files: +```bash +$ git add .github/workflows +$ git commit -m "Manual update to docker workflows" +$ git push origin main +``` -## Deployment +This will automatically start the "01 Maintain: Build and Deploy Site" workflow. -### 01 Build and Deploy (sandpaper-main.yaml) +This will be the extent of requirements for non-renv lessons. -This is the main driver that will only act on the main branch of the repository. -This workflow does the following: +#### Lessons that use Rmd and {renv} + +For renv-enabled lessons: +- Cancel the "01 Maintain: Build and Deploy Site" run that automatically started following the push to main +- Run the "02 Maintain: Check for Updated Packages" +- Run the "03 Maintain: Apply Package Cache" +- Run the "01 Maintain: Build and Deploy Site" + +### Updating using GitHub + +This presumes you: + - already have a lesson repository available on GitHub + - have enabled workflows in the lesson repo + - have set up a SANDPAPER_WORKFLOW personal access token (PAT) in the lesson repo + +To go through these steps, please follow the [Forking a Workbench Lesson](https://docs.carpentries.org/resources/curriculum/lesson-forks.html#forking-a-workbench-lesson-repository) +documentation. + +Once set up, run the "04 Maintain: Update Workflow Files" (`update-workflows.yaml`) action. + +This will raise a PR with any changes to the workflows that are needed. +If you are happy with the changes made, you can merge the PR into your lesson repository. +## Lesson Builds and Deployment + +### 01 Maintain: Build and Deploy Site (docker_build_deploy.yaml) + +This is the main workflow that you will encounter most often. + +It will only act on the main branch of the lesson repository. + +This workflow does the following: 1. checks out the lesson 2. provisions the following resources - - R - - pandoc - - lesson infrastructure (stored in a cache) - - lesson dependencies if needed (stored in a cache) + - the Workbench Docker container + - lesson dependencies if needed (stored in a cache) 3. builds the lesson via `sandpaper:::ci_deploy()` +If your lesson contains rendered content using RMarkdown and/or any associated R package dependencies, you will need to generate and apply the renv cache. +Please read the [Caching](#caching) section below. + #### Caching -This workflow has two caches; one cache is for the lesson infrastructure and -the other is for the lesson dependencies if the lesson contains rendered -content. These caches are invalidated by new versions of the infrastructure and -the `renv.lock` file, respectively. If there is a problem with the cache, -manual invaliation is necessary. You will need maintain access to the repository -and you can either go to the actions tab and [click on the caches button to find -and invalidate the failing cache](https://github.blog/changelog/2022-10-20-manage-caches-in-your-actions-workflows-from-web-interface/) -or by setting the `CACHE_VERSION` secret to the current date (which will -invalidate all of the caches). +> [!NOTE] +> Caching is only relevant for lessons that use Rmd files and renv to manage R packages. +> If you are building basic markdown documents, caching will not apply to you, and the only +> workflow that needs to be run is "01 Maintain: Build and Deploy Site". + +In summary, generating a reusable package cache is achieved by running the "02 Maintain: Check for Updated Packages" workflow, and then the "03 Maintain: Apply Package Cache" workflow. + +These workflows are separated to ensure that once you have a successful build with a working renv cache, this cache is stored within GitHub's infrastructure, and will be reused by the Workbench Docker container. +This means that lesson builds will be faster once an renv cache is created and reused by the Docker container. + +Another major bonus of this setup is that you can keep using this cache indefinitely to build your lesson. +This is important if you need very specific versions of R packages ("pinning"). + +If and when you want to perform an update to the cache, you can re-run the "02 Maintain: Check for Updated Packages" and verify that your lesson still builds with the new packages. +If all looks good, re-run the "03 Maintain: Apply Package Cache" workflow, and this will write a new renv cache file to GitHub. + +In any case, the renv cache is invalidated by new versions of the `renv.lock` file. +This happens: + - if you update your lockfile locally by using the `sandpaper::update_cache()` function, and then push it to the lesson repository + - when you run the "02 Maintain: Check for Updated Packages" and there are new packages to install + +More information on managing local renv caches for lessons can be found in the [Sandpaper packages vignettes](https://carpentries.github.io/sandpaper/articles/building-with-renv.html). + +#### Using different package cache versions + +There are times when you may want to go back to a previous renv package cache file: + - if you run "02 Maintain: Check for Updated Packages" and "03 Maintain: Apply Package Cache" and the cache generation fails for some reason + - if there is a new R package that produces incorrect or broken lesson output + +To choose a previous cache file version for your builds, go to the Actions tab, and click Caches in the left hand pane. + +Cache files should have the following name format: + +``` + OS HASHSUM +[ | ] [ | ] +Linux--renv-2e499eb706112971b2cffceb49b55a6efe49f3ed75cd6579b10ff224489daca4 +``` + +Once you have 2 or more cache files, you can choose which one you want to use. + +Copy the hashsum part of the desired cache file you want to use, e.g. `2e499eb706112971b2cffceb49b55a6efe49f3ed75cd6579b10ff224489daca4`. + +Then either: + 1. Add a repository variable called CACHE_VERSION, and paste in the hash + - Go to ... + 2. Run the "01 Maintain: Build and Deploy Site" manually, supplying the CACHE_VERSION input + - Go to ... + +If you have no caches listed, make sure to run the "02 Maintain: Check for Updated Packages" and "03 Maintain: Apply Package Cache" to create a new renv cache file. ## Updates ### Setup Information -These workflows run on a schedule and at the maintainer's request. Because they -create pull requests that update workflows/require the downstream actions to run, +These workflows run on a mix of schedules, automatic triggers, and at the maintainer's request. +Because they create pull requests that update workflows/require the downstream actions to run, they need a special repository/organization secret token called `SANDPAPER_WORKFLOW` and it must have the `public_repo` and `workflow` scope. This can be an individual user token, OR it can be a trusted bot account. If you -have a repository in one of the official Carpentries accounts, then you do not +have a repository in one of the official Carpentries organisations, then you do not need to worry about this token being present because the Carpentries Core Team will take care of supplying this token. @@ -73,45 +166,69 @@ clipboard and then go to your repository's settings > secrets > actions and create or edit the `SANDPAPER_WORKFLOW` secret, pasting in the generated token. If you do not specify your token correctly, the runs will not fail and they will -give you instructions to provide the token for your repository. +give you instructions to provide the token for your repository. + +### "02 Maintain: Check for Updated Packages" (update-cache.yaml) + +For lessons that have generated content, we use {renv} to ensure that the output +is stable. This is controlled by a single lockfile which documents the packages +needed for the lesson and the version numbers. This workflow is skipped in +lessons that do not have generated content. + +Packages are frequently updated, fixing bugs or introducing new features. It's a +good idea to make sure these packages can be both: updated periodically, or; or left +static to ensure consistent lesson builds. + +The update cache workflow will do this by: +- checking repositories for updates +- updating the renv lockfile +- summarising the updated packages and their versions in a branch called `updates/packages` +- creating a pull request with _only the renv lockfile changed_ + +From here, the markdown documents will be rebuilt and you can inspect what has +changed based on how the packages have updated. -### 02 Maintain: Update Workflow Files (update-workflow.yaml) +If all steps pass in this workflow, you can safely merge the PR that is raised. +Once the PR is merged, the "03 Maintain: Apply Package Cache" workflow will run +automatically. + +### 03 Maintain: Apply Package Cache (docker_apply_cache.yaml) + +This workflow takes the updated lockfile produced in "02 Maintain: Check for Updated Packages" +and uses it to produce a cached file stored within GitHub's infrastructure. + +This cached file can then be reused repeatedly by the "01 Maintain: Build and Deploy Site" +workflow. + +This workflow is run automatically when the PR generated by "02 Maintain: Check for Updated Packages" +is closed and merged. + +You would only ever need to run this workflow manually: +- if your cache gets removed by GitHub due to age or non-use +- if your cache file contains packages that cannot be used by a Workbench Docker container's newer R version + +### "04 Maintain: Update Workflow Files" (update-workflows.yaml) The {sandpaper} repository was designed to do as much as possible to separate -the tools from the content. For local builds, this is absolutely true, but -there is a minor issue when it comes to workflow files: they must live inside -the repository. +the tools from the content. For local builds, this is absolutely true as you +can develop and build lessons without any GitHub workflows. When it comes to +workflow files on GitHub itself for managed builds online, the workflows must +live inside the lesson repository. -This workflow ensures that the workflow files are up-to-date. The way it work is -to download the update-workflows.sh script from GitHub and run it. The script -will do the following: +This workflow ensures that the workflow files are up-to-date. It downloads the +`update-workflows.sh` script from GitHub and runs it. The script will do the +following: -1. check the recorded version of sandpaper against the current version on github +1. check the recorded version of sandpaper against the current version on GitHub 2. update the files if there is a difference in versions -After the files are updated, if there are any changes, they are pushed to a +After the files are updated, and if there are any changes, they are pushed to a branch called `update/workflows` and a pull request is created. Maintainers are encouraged to review the changes and accept the pull request if the outputs are okay. This update is run weekly or on demand. -### 03 Maintain: Update Package Cache (update-cache.yaml) - -For lessons that have generated content, we use {renv} to ensure that the output -is stable. This is controlled by a single lockfile which documents the packages -needed for the lesson and the version numbers. This workflow is skipped in -lessons that do not have generated content. - -Because the lessons need to remain current with the package ecosystem, it's a -good idea to make sure these packages can be updated periodically. The -update cache workflow will do this by checking for updates, applying them in a -branch called `updates/packages` and creating a pull request with _only the -lockfile changed_. - -From here, the markdown documents will be rebuilt and you can inspect what has -changed based on how the packages have updated. - ## Pull Request and Review Management Because our lessons execute code, pull requests are a secruity risk for any @@ -140,11 +257,11 @@ Once the checks are finished, a comment is issued to the pull request, which will allow maintainers to determine if it is safe to run the "Receive Pull Request" workflow from new contributors. -### Receive Pull Request (pr-receive.yaml) +### Receive Pull Request (docker_pr_receive.yaml) **Note of caution:** This workflow runs arbitrary code by anyone who creates a pull request. GitHub has safeguarded the token used in this workflow to have no -priviledges in the repository, but we have taken precautions to protect against +privileges in the repository, but we have taken precautions to protect against spoofing. This workflow is triggered with every push to a pull request. If this workflow @@ -164,14 +281,11 @@ request. This builds the content and uploads three artifacts: 2. A summary of changes after the rendering process (diff) 3. The rendered files (build) -Because this workflow builds generated content, it follows the same general -process as the `sandpaper-main` workflow with the same caching mechanisms. - -The artifacts produced are used by the next workflow. +The artifacts produced are used by the "Comment on Pull Request" workflow. ### Comment on Pull Request (pr-comment.yaml) -This workflow is triggered if the `pr-receive.yaml` workflow is successful. +This workflow is triggered if the `docker_pr_receive.yaml` workflow is successful. The steps in this workflow are: 1. Test if the workflow is valid and comment the validity of the workflow to the diff --git a/.github/workflows/docker_apply_cache.yaml b/.github/workflows/docker_apply_cache.yaml new file mode 100644 index 00000000..0cb66370 --- /dev/null +++ b/.github/workflows/docker_apply_cache.yaml @@ -0,0 +1,227 @@ +name: "03 Maintain: Apply Package Cache" +description: "Generate the package cache for the lesson after a pull request has been merged or via manual trigger, and cache in S3 or GitHub" +on: + workflow_dispatch: + inputs: + name: + description: 'Who triggered this build?' + required: true + default: 'Maintainer (via GitHub)' + pull_request: + types: + - closed + branches: + - main + +# queue cache runs +concurrency: + group: docker-apply-cache + cancel-in-progress: false + +jobs: + preflight: + name: "Preflight: PR or Manual Trigger?" + runs-on: ubuntu-latest + outputs: + do-apply: ${{ steps.check.outputs.merged_or_manual }} + steps: + - name: "Should we run cache application?" + id: check + run: | + if [[ "${{ github.event_name }}" == "workflow_dispatch" || + ("${{ github.ref }}" == "refs/heads/main" && "${{ github.event.action }}" == "closed" && "${{ github.event.pull_request.merged }}" == "true") ]]; then + echo "merged_or_manual=true" >> $GITHUB_OUTPUT + else + echo "This was not a manual trigger and no PR was merged. No action taken." + echo "merged_or_manual=false" >> $GITHUB_OUTPUT + fi + shell: bash + + check-renv: + name: "Check If We Need {renv}" + runs-on: ubuntu-latest + needs: preflight + if: needs.preflight.outputs.do-apply == 'true' + permissions: + id-token: write + outputs: + renv-needed: ${{ steps.check-for-renv.outputs.renv-needed }} + renv-cache-hashsum: ${{ steps.check-for-renv.outputs.renv-cache-hashsum }} + renv-cache-available: ${{ steps.check-for-renv.outputs.renv-cache-available }} + steps: + - name: "Check for renv" + id: check-for-renv + uses: carpentries/actions/renv-checks@main + with: + role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }} + aws-region: ${{ secrets.AWS_GH_OIDC_REGION }} + WORKBENCH_TAG: ${{ vars.WORKBENCH_TAG || 'latest' }} + token: ${{ secrets.GITHUB_TOKEN }} + + no-renv-cache-used: + name: "No renv cache used" + runs-on: ubuntu-latest + needs: check-renv + if: needs.check-renv.outputs.renv-needed != 'true' + steps: + - name: "No renv cache needed" + run: echo "No renv cache needed for this lesson" + + renv-cache-available: + name: "renv cache available" + runs-on: ubuntu-latest + needs: check-renv + if: needs.check-renv.outputs.renv-cache-available == 'true' + steps: + - name: "renv cache available" + run: echo "renv cache available for this lesson" + + update-renv-cache: + name: "Update renv Cache" + runs-on: ubuntu-latest + needs: check-renv + if: | + needs.check-renv.outputs.renv-needed == 'true' && + needs.check-renv.outputs.renv-cache-available != 'true' && + ( + github.event_name == 'workflow_dispatch' || + ( + github.event.pull_request.merged == true && + ( + ( + contains( + join(github.event.pull_request.labels.*.name, ','), + 'type: package cache' + ) && + github.event.pull_request.head.ref == 'update/packages' + ) + || + ( + contains( + join(github.event.pull_request.labels.*.name, ','), + 'type: workflows' + ) && + github.event.pull_request.head.ref == 'update/workflows' + ) + || + ( + contains( + join(github.event.pull_request.labels.*.name, ','), + 'type: docker version' + ) && + github.event.pull_request.head.ref == 'update/workbench-docker-version' + ) + ) + ) + ) + permissions: + checks: write + contents: write + pages: write + id-token: write + container: + image: ghcr.io/carpentries/workbench-docker:${{ vars.WORKBENCH_TAG || 'latest' }} + env: + WORKBENCH_PROFILE: "ci" + GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} + RENV_PATHS_ROOT: /home/rstudio/lesson/renv + RENV_PROFILE: "lesson-requirements" + RENV_VERSION: ${{ needs.check-renv.outputs.renv-cache-hashsum }} + RENV_CONFIG_EXTERNAL_LIBRARIES: "/usr/local/lib/R/site-library" + volumes: + - ${{ github.workspace }}:/home/rstudio/lesson + options: --cpus 2 + steps: + - uses: actions/checkout@v4 + + - name: "Debugging Info" + run: | + echo "Current Directory: $(pwd)" + ls -lah /home/rstudio/.workbench + ls -lah $(pwd) + Rscript -e 'sessionInfo()' + shell: bash + + - name: "Mark Repository as Safe" + run: | + git config --global --add safe.directory $(pwd) + shell: bash + + - name: "Ensure sandpaper is loadable" + run: | + .libPaths() + library(sandpaper) + shell: Rscript {0} + + - name: "Setup Lesson Dependencies" + run: | + Rscript /home/rstudio/.workbench/setup_lesson_deps.R + shell: bash + + - name: "Fortify renv Cache" + run: | + Rscript /home/rstudio/.workbench/fortify_renv_cache.R + shell: bash + + - name: "Get Container Version Used" + id: wb-vers + uses: carpentries/actions/container-version@main + with: + WORKBENCH_TAG: ${{ vars.WORKBENCH_TAG }} + renv-needed: ${{ needs.check-renv.outputs.renv-needed }} + token: ${{ secrets.GITHUB_TOKEN }} + + - name: "Validate Current Org and Workflow" + id: validate-org-workflow + uses: carpentries/actions/validate-org-workflow@main + with: + repo: ${{ github.repository }} + workflow: ${{ github.workflow }} + + - name: "Configure AWS credentials via OIDC" + id: aws-creds + env: + role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }} + aws-region: ${{ secrets.AWS_GH_OIDC_REGION }} + if: | + steps.validate-org-workflow.outputs.is_valid == 'true' && + env.role-to-assume != '' && + env.aws-region != '' + uses: aws-actions/configure-aws-credentials@v5.0.0 + with: + role-to-assume: ${{ env.role-to-assume }} + aws-region: ${{ env.aws-region }} + output-credentials: true + + - name: "Upload cache object to S3" + id: upload-cache + uses: carpentries/actions-cache@frog-matchedkey-1 + with: + accessKey: ${{ steps.aws-creds.outputs.aws-access-key-id }} + secretKey: ${{ steps.aws-creds.outputs.aws-secret-access-key }} + sessionToken: ${{ steps.aws-creds.outputs.aws-session-token }} + bucket: workbench-docker-caches + path: | + /home/rstudio/lesson/renv + /usr/local/lib/R/site-library + key: ${{ github.repository }}/${{ steps.wb-vers.outputs.container-version }}_renv-${{ needs.check-renv.outputs.renv-cache-hashsum }} + restore-keys: + ${{ github.repository }}/${{ steps.wb-vers.outputs.container-version }}_renv- + + trigger-build-deploy: + name: "Trigger Build and Deploy Workflow" + runs-on: ubuntu-latest + needs: update-renv-cache + if: | + needs.update-renv-cache.result == 'success' || + needs.check-renv.outputs.renv-cache-available == 'true' + steps: + - uses: actions/checkout@v4 + + - name: "Trigger Build and Deploy Workflow" + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + gh workflow run docker_build_deploy.yaml --ref main + shell: bash + continue-on-error: true diff --git a/.github/workflows/docker_build_deploy.yaml b/.github/workflows/docker_build_deploy.yaml new file mode 100644 index 00000000..c14f776c --- /dev/null +++ b/.github/workflows/docker_build_deploy.yaml @@ -0,0 +1,155 @@ +name: "01 Maintain: Build and Deploy Site" +description: "Build and deploy the lesson site using the carpentries/workbench-docker container" +on: + push: + branches: + - 'main' + paths-ignore: + - '.github/workflows/**.yaml' + - '.github/workbench-docker-version.txt' + schedule: + - cron: '0 0 * * 2' + workflow_dispatch: + inputs: + name: + description: 'Who triggered this build?' + required: true + default: 'Maintainer (via GitHub)' + CACHE_VERSION: + description: 'Optional renv cache version override' + required: false + default: '' + reset: + description: 'Reset cached markdown files' + required: true + default: false + type: boolean + force-skip-manage-deps: + description: 'Skip build-time dependency management' + required: true + default: false + type: boolean + +# only one build/deploy at a time +concurrency: + group: docker-build-deploy + cancel-in-progress: true + +jobs: + preflight: + name: "Preflight: Schedule, Push, or PR?" + runs-on: ubuntu-latest + outputs: + do-build: ${{ steps.build-check.outputs.do-build }} + renv-needed: ${{ steps.build-check.outputs.renv-needed }} + renv-cache-hashsum: ${{ steps.build-check.outputs.renv-cache-hashsum }} + workbench-container-file-exists: ${{ steps.wb-vers.outputs.workbench-container-file-exists }} + wb-vers: ${{ steps.wb-vers.outputs.container-version }} + last-wb-vers: ${{ steps.wb-vers.outputs.last-container-version }} + workbench-update: ${{ steps.wb-vers.outputs.workbench-update }} + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + steps: + - name: "Should we run build and deploy?" + id: build-check + uses: carpentries/actions/build-preflight@main + + - name: "Checkout Lesson" + if: steps.build-check.outputs.do-build == 'true' + uses: actions/checkout@v4 + + - name: "Get container version info" + id: wb-vers + if: steps.build-check.outputs.do-build == 'true' + uses: carpentries/actions/container-version@main + with: + WORKBENCH_TAG: ${{ vars.WORKBENCH_TAG }} + renv-needed: ${{ steps.build-check.outputs.renv-needed }} + token: ${{ secrets.GITHUB_TOKEN }} + + full-build: + name: "Build Full Site" + runs-on: ubuntu-latest + needs: preflight + if: | + always() && + needs.preflight.outputs.do-build == 'true' && + needs.preflight.outputs.workbench-update != 'true' + env: + RENV_EXISTS: ${{ needs.preflight.outputs.renv-needed }} + RENV_HASH: ${{ needs.preflight.outputs.renv-cache-hashsum }} + permissions: + checks: write + contents: write + pages: write + id-token: write + container: + image: ghcr.io/carpentries/workbench-docker:${{ vars.WORKBENCH_TAG || 'latest' }} + env: + WORKBENCH_PROFILE: "ci" + GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} + RENV_PATHS_ROOT: /home/rstudio/lesson/renv + RENV_PROFILE: "lesson-requirements" + RENV_CONFIG_EXTERNAL_LIBRARIES: "/usr/local/lib/R/site-library" + volumes: + - ${{ github.workspace }}:/home/rstudio/lesson + options: --cpus 1 + steps: + - uses: actions/checkout@v4 + + - name: "Debugging Info" + run: | + cd /home/rstudio/lesson + echo "Current Directory: $(pwd)" + echo "RENV_HASH is $RENV_HASH" + ls -lah /home/rstudio/.workbench + ls -lah $(pwd) + Rscript -e 'sessionInfo()' + shell: bash + + - name: "Mark Repository as Safe" + run: | + git config --global --add safe.directory $(pwd) + shell: bash + + - name: "Setup Lesson Dependencies" + id: build-container-deps + uses: carpentries/actions/build-container-deps@main + with: + CACHE_VERSION: ${{ vars.CACHE_VERSION || github.event.inputs.CACHE_VERSION || '' }} + WORKBENCH_TAG: ${{ vars.WORKBENCH_TAG || 'latest' }} + LESSON_PATH: ${{ vars.LESSON_PATH || '/home/rstudio/lesson' }} + role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }} + aws-region: ${{ secrets.AWS_GH_OIDC_REGION }} + token: ${{ secrets.GITHUB_TOKEN }} + + - name: "Run Container and Build Site" + id: build-and-deploy + uses: carpentries/actions/build-and-deploy@main + with: + reset: ${{ github.event.inputs.reset || 'false' }} + skip-manage-deps: ${{ github.event.inputs.force-skip-manage-deps == 'true' || steps.build-container-deps.outputs.renv-cache-available || steps.build-container-deps.outputs.backup-cache-used || 'false' }} + + update-container-version: + name: "Update container version used" + runs-on: ubuntu-latest + needs: [preflight] + permissions: + actions: write + contents: write + pull-requests: write + id-token: write + if: | + needs.preflight.outputs.do-build == 'true' && + ( + needs.preflight.outputs.workbench-container-file-exists == 'false' || + needs.preflight.outputs.workbench-update == 'true' + ) + steps: + - name: "Record container version used" + uses: carpentries/actions/record-container-version@main + with: + CONTAINER_VER: ${{ needs.preflight.outputs.wb-vers }} + token: ${{ secrets.GITHUB_TOKEN }} + role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }} + aws-region: ${{ secrets.AWS_GH_OIDC_REGION }} diff --git a/.github/workflows/docker_pr_receive.yaml b/.github/workflows/docker_pr_receive.yaml new file mode 100644 index 00000000..12b16bf7 --- /dev/null +++ b/.github/workflows/docker_pr_receive.yaml @@ -0,0 +1,302 @@ +name: "Bot: Receive Pull Request" +description: "Receive a pull request and build the markdown source files" +on: + pull_request: + types: + [opened, synchronize, reopened] + workflow_dispatch: + inputs: + pr_number: + type: number + required: true + +concurrency: + group: ${{ github.ref }} + cancel-in-progress: true + +permissions: + contents: read + pull-requests: write + +jobs: + + preflight: + name: "Preflight: md-outputs exists?" + runs-on: ubuntu-latest + outputs: + branch-exists: ${{ steps.check.outputs.exists }} + steps: + - name: "Checkout Lesson" + uses: actions/checkout@v4 + + - name: "Check if md-outputs branch exists" + id: check + run: | + # 💡 Checking for md-outputs branch # + if [[ -n $(git ls-remote --exit-code --heads origin md-outputs) ]]; then + echo "exists=true" >> $GITHUB_OUTPUT + else + echo "exists=false" >> $GITHUB_OUTPUT + echo "::error::md-outputs branch required but does not exist." + echo "::error::Please merge any open package update PRs to trigger the '03 Maintain: Apply Package Cache' and '01: Maintain: Build and Deploy Site' workflows." + + echo "## ❌ ERROR: md-outputs branch required" >> $GITHUB_STEP_SUMMARY + echo "Please merge any open package update PRs to trigger the '03 Maintain: Apply Package Cache' and '01: Maintain: Build and Deploy Site' workflows." >> $GITHUB_STEP_SUMMARY + + exit 1 + fi + shell: bash + + test-pr: + name: "Record PR number" + if: ${{ github.event.action != 'closed' }} && ${{ needs.preflight.outputs.branch-exists == 'true' }} + runs-on: ubuntu-latest + needs: preflight + outputs: + is_valid: ${{ steps.check-pr.outputs.VALID }} + pr_number: ${{ env.NR }} + pr_branch: ${{ env.PR_BRANCH }} + steps: + - name: "Grab PR" + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + if [[ "${{ github.event_name }}" == "pull_request" ]] ; then + PR_NUMBER=${{ github.event.number }} + elif [[ "${{ github.event_name }}" == "workflow_dispatch" ]] ; then + PR_NUMBER=${{ inputs.pr_number }} + fi + + echo $PR_NUMBER > ${{ github.workspace }}/NR + echo "NR=$PR_NUMBER" >> $GITHUB_ENV + echo "PR_BRANCH=$(gh -R ${{ github.repository }} pr view $PR_NUMBER --json headRefName --jq '.headRefName')" >> $GITHUB_ENV + shell: bash + + - name: "Upload PR number" + id: upload + if: always() + uses: actions/upload-artifact@v4 + with: + name: pr + path: ${{ github.workspace }}/NR + + - name: "Get Invalid Hashes File" + id: hash + run: | + echo "json<> $GITHUB_OUTPUT + shell: bash + + - name: "Debug Hashes Output" + run: | + echo "${{ steps.hash.outputs.json }}" + shell: bash + + - name: "Check PR" + id: check-pr + uses: carpentries/actions/check-valid-pr@main + with: + pr: ${{ env.NR }} + invalid: ${{ fromJSON(steps.hash.outputs.json)[github.repository] }} + + check-renv: + name: "Check If We Need {renv}" + runs-on: ubuntu-latest + outputs: + renv-needed: ${{ steps.renv-check.outputs.renv-needed }} + renv-cache-hashsum: ${{ steps.renv-check.outputs.renv-cache-hashsum }} + steps: + - name: "Checkout Lesson" + uses: actions/checkout@v4 + + - name: "Is renv required?" + id: renv-check + uses: carpentries/actions/renv-checks@main + with: + CACHE_VERSION: ${{ inputs.CACHE_VERSION || '' }} + skip-cache-check: true + + build-md-source: + name: "Build markdown source files if valid" + needs: + - test-pr + - check-renv + runs-on: ubuntu-latest + if: needs.test-pr.outputs.is_valid == 'true' + env: + CHIVE: ${{ github.workspace }}/site/chive + PR: ${{ github.workspace }}/site/pr + GHWMD: ${{ github.workspace }}/site/built + PR_BRANCH: ${{ needs.test-pr.outputs.pr_branch }} + PR_NUMBER: ${{ needs.test-pr.outputs.pr_number }} + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + permissions: + checks: write + contents: write + pages: write + container: + image: ghcr.io/carpentries/workbench-docker:${{ vars.WORKBENCH_TAG || 'latest' }} + env: + WORKBENCH_PROFILE: "ci" + GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + RENV_PATHS_ROOT: /home/rstudio/lesson/renv + RENV_PROFILE: "lesson-requirements" + RENV_CONFIG_EXTERNAL_LIBRARIES: "/usr/local/lib/R/site-library" + volumes: + - ${{ github.workspace }}:/home/rstudio/lesson + options: --cpus 2 + outputs: + workbench-update: ${{ steps.wb-vers.outputs.workbench-update }} + build-site: ${{ steps.build-site.outcome }} + steps: + - uses: actions/checkout@v4 + + - name: "Check Out Staging Branch" + uses: actions/checkout@v4 + with: + ref: md-outputs + path: ${{ env.GHWMD }} + + - name: Mark Repository as Safe + run: | + git config --global --add safe.directory $(pwd) + git config --global --add safe.directory /home/rstudio/lesson + shell: bash + + - name: "Ensure sandpaper is loadable" + run: | + .libPaths() + library(sandpaper) + shell: Rscript {0} + + - name: Setup Lesson Dependencies + run: | + Rscript /home/rstudio/.workbench/setup_lesson_deps.R + shell: bash + + - name: Get Container Version Used + id: wb-vers + if: needs.check-renv.outputs.renv-needed == 'true' + uses: carpentries/actions/container-version@main + with: + WORKBENCH_TAG: ${{ vars.WORKBENCH_TAG }} + renv-needed: ${{ needs.check-renv.outputs.renv-needed }} + token: ${{ secrets.GITHUB_TOKEN }} + + - name: "Validate Current Org and Workflow" + id: validate-org-workflow + if: needs.check-renv.outputs.renv-needed == 'true' + uses: carpentries/actions/validate-org-workflow@main + with: + repo: ${{ github.repository }} + workflow: ${{ github.workflow }} + + - name: Configure AWS credentials via OIDC + id: aws-creds + env: + role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }} + aws-region: ${{ secrets.AWS_GH_OIDC_REGION }} + if: | + steps.validate-org-workflow.outputs.is_valid == 'true' && + needs.check-renv.outputs.renv-needed == 'true' && + env.role-to-assume != '' && + env.aws-region != '' + uses: aws-actions/configure-aws-credentials@v5.0.0 + with: + role-to-assume: ${{ env.role-to-assume }} + aws-region: ${{ env.aws-region }} + output-credentials: true + + - name: Get cache object from S3 + id: s3-cache + uses: carpentries/actions-cache/restore@frog-matchedkey-1 + if: needs.check-renv.outputs.renv-needed == 'true' + with: + # insecure: false # optional, use http instead of https. default false + accessKey: ${{ steps.aws-creds.outputs.aws-access-key-id }} + secretKey: ${{ steps.aws-creds.outputs.aws-secret-access-key }} + sessionToken: ${{ steps.aws-creds.outputs.aws-session-token }} + bucket: workbench-docker-caches + path: | + /home/rstudio/lesson/renv + /usr/local/lib/R/site-library + key: ${{ github.repository }}/${{ steps.wb-vers.outputs.container-version }}_renv-${{ needs.check-renv.outputs.renv-cache-hashsum }} + restore-keys: + ${{ github.repository }}/${{ steps.wb-vers.outputs.container-version }}_renv- + + - name: "Fortify renv Cache" + if: | + needs.check-renv.outputs.renv-needed == 'true' && + steps.s3-cache.outputs.cache-hit != 'true' + run: | + Rscript /home/rstudio/.workbench/fortify_renv_cache.R + shell: bash + + - name: "Validate and Build Markdown" + id: build-site + run: | + sandpaper::package_cache_trigger(TRUE) + sandpaper::validate_lesson(path = '/home/rstudio/lesson') + sandpaper:::build_markdown(path = '/home/rstudio/lesson', quiet = FALSE) + shell: Rscript {0} + + - name: "Generate Artifacts" + id: generate-artifacts + run: | + sandpaper:::ci_bundle_pr_artifacts( + repo = '${{ github.repository }}', + pr_number = '${{ env.PR_NUMBER }}', + path_md = '/home/rstudio/lesson/site/built', + path_pr = '/home/rstudio/lesson/site/pr', + path_archive = '/home/rstudio/lesson/site/chive', + branch = 'md-outputs' + ) + shell: Rscript {0} + + - name: "Upload PR" + uses: actions/upload-artifact@v4 + with: + name: pr + path: ${{ env.PR }} + overwrite: true + + - name: "Upload Diff" + uses: actions/upload-artifact@v4 + with: + name: diff + path: ${{ env.CHIVE }} + retention-days: 1 + + - name: "Upload Build" + uses: actions/upload-artifact@v4 + with: + name: built + path: ${{ env.GHWMD }} + retention-days: 1 + + - name: "Teardown" + run: sandpaper::reset_site() + shell: Rscript {0} + + pr-checks: + name: "Trigger PR Checks?" + needs: + - test-pr + - build-md-source + runs-on: ubuntu-latest + if: needs.test-pr.outputs.is_valid == 'true' + permissions: + actions: write + checks: write + steps: + - name: "Checkout Lesson" + uses: actions/checkout@v4 + + - name: "Trigger PR Checks" + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + gh workflow run pr-comment.yaml --ref main --field workflow_id=${{ github.run_id }} + shell: bash diff --git a/.github/workflows/pr-comment.yaml b/.github/workflows/pr-comment.yaml index f80d9d0c..cbf0e2b2 100755 --- a/.github/workflows/pr-comment.yaml +++ b/.github/workflows/pr-comment.yaml @@ -1,40 +1,44 @@ name: "Bot: Comment on the Pull Request" - -# read-write repo token -# access to secrets +description: "Comment on the pull request with the results of the markdown generation" on: - workflow_run: - workflows: ["Receive Pull Request"] - types: - - completed + workflow_dispatch: + inputs: + workflow_id: + required: true concurrency: group: pr-${{ github.event.workflow_run.pull_requests[0].number }} cancel-in-progress: true - jobs: # Pull requests are valid if: # - they match the sha of the workflow run head commit # - they are open - # - no .github files were committed + # - no .github files were committed, except for .github/workbench-docker-version.txt test-pr: name: "Test if pull request is valid" - runs-on: ubuntu-22.04 + runs-on: ubuntu-latest if: > - github.event.workflow_run.event == 'pull_request' && - github.event.workflow_run.conclusion == 'success' + github.event_name == 'workflow_dispatch' || + ( + github.event_name == 'workflow_run' && + ( + github.event.workflow_run.event == 'pull_request' || + github.event.workflow_run.event == 'workflow_dispatch' + ) && + github.event.workflow_run.conclusion == 'success' + ) outputs: is_valid: ${{ steps.check-pr.outputs.VALID }} payload: ${{ steps.check-pr.outputs.payload }} number: ${{ steps.get-pr.outputs.NUM }} msg: ${{ steps.check-pr.outputs.MSG }} steps: - - name: 'Download PR artifact' + - name: "Download PR artifact" id: dl uses: carpentries/actions/download-workflow-artifact@main with: - run: ${{ github.event.workflow_run.id }} + run: ${{ github.event.workflow_run.id || inputs.workflow_id }} name: 'pr' - name: "Get PR Number" @@ -50,12 +54,46 @@ jobs: run: | echo '::error::A pull request number was not recorded. The pull request that triggered this workflow is likely malicious.' exit 1 + + - name: "Checkout Lesson" + uses: actions/checkout@v4 + + - name: "Verify committed files" + id: changed-files + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + ## Get list of changed files in the PR ## + ONLY_VERSION=$(gh pr view ${{ steps.get-pr.outputs.NUM }} --json files --jq ' + .files | + length == 1 and + .[0].path == ".github/workbench-docker-version.txt" + ') + + if [[ "$ONLY_VERSION" == "true" ]]; then + echo "only_version_file=true" >> $GITHUB_OUTPUT + else + echo "only_version_file=false" >> $GITHUB_OUTPUT + fi + shell: bash + + - name: "Skip checks for Workbench version file updates" + if: steps.changed-files.outputs.only_version_file == 'true' + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + echo "Only workbench-docker-version.txt changed, skipping preflight checks and running cache update" + gh workflow run update-cache.yaml --ref main + exit 0 + shell: bash + - name: "Get Invalid Hashes File" id: hash run: | echo "json<> $GITHUB_OUTPUT + - name: "Check PR" id: check-pr if: ${{ steps.dl.outputs.success == 'true' }} @@ -67,6 +105,14 @@ jobs: invalid: ${{ fromJSON(steps.hash.outputs.json)[github.repository] }} fail_on_error: true + - name: "Comment result of validation" + id: comment-diff + if: always() + uses: carpentries/actions/comment-diff@main + with: + pr: ${{ steps.get-pr.outputs.NUM }} + body: ${{ steps.check-pr.outputs.MSG }} + # Create an orphan branch on this repository with two commits # - the current HEAD of the md-outputs branch # - the output from running the current HEAD of the pull request through @@ -74,32 +120,32 @@ jobs: create-branch: name: "Create Git Branch" needs: test-pr - runs-on: ubuntu-22.04 - if: ${{ needs.test-pr.outputs.is_valid == 'true' }} + runs-on: ubuntu-latest + if: needs.test-pr.outputs.is_valid == 'true' env: NR: ${{ needs.test-pr.outputs.number }} permissions: contents: write steps: - - name: 'Checkout md outputs' + - name: "Checkout md outputs" uses: actions/checkout@v4 with: ref: md-outputs path: built fetch-depth: 1 - - name: 'Download built markdown' + - name: "Download built markdown" id: dl uses: carpentries/actions/download-workflow-artifact@main with: - run: ${{ github.event.workflow_run.id }} + run: ${{ github.event.workflow_run.id || inputs.workflow_id }} name: 'built' - - if: ${{ steps.dl.outputs.success == 'true' }} + - if: steps.dl.outputs.success == 'true' run: unzip built.zip - name: "Create orphan and push" - if: ${{ steps.dl.outputs.success == 'true' }} + if: steps.dl.outputs.success == 'true' run: | cd built/ git config --local user.email "actions@github.com" @@ -120,26 +166,26 @@ jobs: comment-pr: name: "Comment on Pull Request" needs: [test-pr, create-branch] - runs-on: ubuntu-22.04 - if: ${{ needs.test-pr.outputs.is_valid == 'true' }} + runs-on: ubuntu-latest + if: needs.test-pr.outputs.is_valid == 'true' env: NR: ${{ needs.test-pr.outputs.number }} permissions: pull-requests: write steps: - - name: 'Download comment artifact' + - name: "Download comment artifact" id: dl uses: carpentries/actions/download-workflow-artifact@main with: - run: ${{ github.event.workflow_run.id }} + run: ${{ github.event.workflow_run.id || inputs.workflow_id }} name: 'diff' - - if: ${{ steps.dl.outputs.success == 'true' }} + - if: steps.dl.outputs.success == 'true' run: unzip ${{ github.workspace }}/diff.zip - name: "Comment on PR" id: comment-diff - if: ${{ steps.dl.outputs.success == 'true' }} + if: steps.dl.outputs.success == 'true' uses: carpentries/actions/comment-diff@main with: pr: ${{ env.NR }} @@ -150,24 +196,26 @@ jobs: comment-changed-workflow: name: "Comment if workflow files have changed" needs: test-pr - runs-on: ubuntu-22.04 - if: ${{ always() && needs.test-pr.outputs.is_valid == 'false' }} + runs-on: ubuntu-latest + if: | + always() && + needs.test-pr.outputs.is_valid == 'false' env: - NR: ${{ github.event.workflow_run.pull_requests[0].number }} + NR: ${{ needs.test-pr.outputs.number }} body: ${{ needs.test-pr.outputs.msg }} permissions: pull-requests: write steps: - - name: 'Check for spoofing' + - name: "Check for spoofing" id: dl uses: carpentries/actions/download-workflow-artifact@main with: - run: ${{ github.event.workflow_run.id }} + run: ${{ github.event.workflow_run.id || inputs.workflow_id }} name: 'built' - - name: 'Alert if spoofed' + - name: "Alert if spoofed" id: spoof - if: ${{ steps.dl.outputs.success == 'true' }} + if: steps.dl.outputs.success == 'true' run: | echo 'body<> $GITHUB_ENV echo '' >> $GITHUB_ENV diff --git a/.github/workflows/pr-preflight.yaml b/.github/workflows/pr-preflight.yaml index 34ad7aed..d0d7420d 100755 --- a/.github/workflows/pr-preflight.yaml +++ b/.github/workflows/pr-preflight.yaml @@ -11,7 +11,7 @@ jobs: test-pr: name: "Test if pull request is valid" if: ${{ github.event.action != 'closed' }} - runs-on: ubuntu-22.04 + runs-on: ubuntu-latest outputs: is_valid: ${{ steps.check-pr.outputs.VALID }} permissions: diff --git a/.github/workflows/sandpaper-version.txt b/.github/workflows/sandpaper-version.txt index ea98690d..267d7e01 100644 --- a/.github/workflows/sandpaper-version.txt +++ b/.github/workflows/sandpaper-version.txt @@ -1 +1 @@ -0.16.12 +0.18.3 diff --git a/.github/workflows/update-cache.yaml b/.github/workflows/update-cache.yaml index a011c0c0..27b6d1cd 100755 --- a/.github/workflows/update-cache.yaml +++ b/.github/workflows/update-cache.yaml @@ -1,26 +1,45 @@ -name: "03 Maintain: Update Package Cache" - +name: "02 Maintain: Check for Updated Packages" +description: "Check for updated R packages and create a pull request to update the lesson's renv lockfile and package cache" on: + schedule: + - cron: '0 0 * * 2' workflow_dispatch: inputs: name: - description: 'Who triggered this build (enter github username to tag yourself)?' + description: 'Who triggered this build?' required: true - default: 'monthly run' - schedule: - # Run every tuesday - - cron: '0 0 * * 2' + default: 'Maintainer (via GitHub)' + force-renv-init: + description: 'Force full lockfile update?' + required: false + default: false + type: boolean + update-packages: + description: 'Install any package updates?' + required: false + default: true + type: boolean + generate-cache: + description: 'Generate separate package cache?' + required: false + default: false + type: boolean + +env: + LOCKFILE_CACHE_GEN: ${{ vars.LOCKFILE_CACHE_GEN || github.event.inputs.generate-cache || 'false' }} + FORCE_RENV_INIT: ${{ vars.FORCE_RENV_INIT || github.event.inputs.force-renv-init || 'false' }} + UPDATE_PACKAGES: ${{ vars.UPDATE_PACKAGES || github.event.inputs.update-packages || 'true' }} jobs: preflight: - name: "Preflight Check" - runs-on: ubuntu-22.04 + name: "Preflight: Manual or Scheduled Trigger?" + runs-on: ubuntu-latest outputs: ok: ${{ steps.check.outputs.ok }} steps: - id: check run: | - if [[ ${{ github.event_name }} == 'workflow_dispatch' ]]; then + if [[ "${{ github.event_name }}" == 'workflow_dispatch' ]]; then echo "ok=true" >> $GITHUB_OUTPUT echo "Running on request" # using single brackets here to avoid 08 being interpreted as octal @@ -33,48 +52,42 @@ jobs: echo "ok=false" >> $GITHUB_OUTPUT echo "Not Running Today" fi + shell: bash - check_renv: - name: "Check if We Need {renv}" - runs-on: ubuntu-22.04 + check-renv: + name: "Check If We Need {renv}" + runs-on: ubuntu-latest needs: preflight - if: ${{ needs.preflight.outputs.ok == 'true'}} + if: ${{ needs.preflight.outputs.ok == 'true' }} outputs: - needed: ${{ steps.renv.outputs.exists }} + renv-needed: ${{ steps.renv-check.outputs.renv-needed }} steps: - name: "Checkout Lesson" uses: actions/checkout@v4 - - id: renv - run: | - if [[ -d renv ]]; then - echo "exists=true" >> $GITHUB_OUTPUT - fi - check_token: - name: "Check SANDPAPER_WORKFLOW token" - runs-on: ubuntu-22.04 - needs: check_renv - if: ${{ needs.check_renv.outputs.needed == 'true' }} - outputs: - workflow: ${{ steps.validate.outputs.wf }} - repo: ${{ steps.validate.outputs.repo }} - steps: - - name: "validate token" - id: validate - uses: carpentries/actions/check-valid-credentials@main + - name: "Is renv required?" + id: renv-check + uses: carpentries/actions/renv-checks@main with: - token: ${{ secrets.SANDPAPER_WORKFLOW }} + CACHE_VERSION: ${{ inputs.CACHE_VERSION || '' }} + skip-cache-check: true update_cache: - name: "Update Package Cache" - needs: check_token - if: ${{ needs.check_token.outputs.repo== 'true' }} + name: "Create Package Update Pull Request" runs-on: ubuntu-22.04 + needs: check-renv + permissions: + contents: write + pull-requests: write + actions: write + issues: write + id-token: write + if: needs.check-renv.outputs.renv-needed == 'true' env: GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} RENV_PATHS_ROOT: ~/.local/share/renv/ steps: - - name: "Checkout Lesson" uses: actions/checkout@v4 @@ -88,14 +101,60 @@ jobs: id: update uses: carpentries/actions/update-lockfile@main with: + update: ${{ env.UPDATE_PACKAGES }} + force-renv-init: ${{ env.FORCE_RENV_INIT }} + generate-cache: ${{ env.LOCKFILE_CACHE_GEN }} cache-version: ${{ secrets.CACHE_VERSION }} - - name: Create Pull Request + - name: "Validate Current Org and Workflow" + id: validate-org-workflow + uses: carpentries/actions/validate-org-workflow@main + with: + repo: ${{ github.repository }} + workflow: ${{ github.workflow }} + + - name: "Configure AWS credentials via OIDC" + env: + role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }} + aws-region: ${{ secrets.AWS_GH_OIDC_REGION }} + if: | + steps.validate-org-workflow.outputs.is_valid == 'true' && + env.role-to-assume != '' && + env.aws-region != '' + uses: aws-actions/configure-aws-credentials@v5.0.0 + with: + role-to-assume: ${{ env.role-to-assume }} + aws-region: ${{ env.aws-region }} + + - name: "Set PAT from AWS Secrets Manager" + env: + role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }} + aws-region: ${{ secrets.AWS_GH_OIDC_REGION }} + if: | + steps.validate-org-workflow.outputs.is_valid == 'true' && + env.role-to-assume != '' && + env.aws-region != '' + id: set-pat + run: | + SECRET=$(aws secretsmanager get-secret-value \ + --secret-id carpentries-bot/github-pat \ + --query SecretString --output text) + PAT=$(echo "$SECRET" | jq -r .[]) + echo "::add-mask::$PAT" + echo "pat=$PAT" >> "$GITHUB_OUTPUT" + shell: bash + + # Create the PR with the following roles in order of preference: + # - Carpentries Bot classic PAT fetched from AWS (will only work in official Carpentries repos) + # - repo-scoped SANDPAPER_WORKFLOW classic PAT (will work in all scenarios) + # - default GITHUB_TOKEN (will work suitably, but workflows need to be triggered) + - name: "Create Pull Request" id: cpr - if: ${{ steps.update.outputs.n > 0 }} + if: | + steps.update.outputs.n > 0 uses: carpentries/create-pull-request@main with: - token: ${{ secrets.SANDPAPER_WORKFLOW }} + token: ${{ steps.set-pat.outputs.pat || secrets.SANDPAPER_WORKFLOW || secrets.GITHUB_TOKEN }} delete-branch: true branch: "update/packages" commit-message: "[actions] update ${{ steps.update.outputs.n }} packages" @@ -123,3 +182,19 @@ jobs: [1]: https://github.com/carpentries/create-pull-request/tree/main labels: "type: package cache" draft: false + + - name: "Skip PR creation" + if: steps.update.outputs.n == 0 + run: | + echo "No updates needed, skipping PR creation" + shell: bash + + # thanks @Bisaloo! - https://github.com/carpentries/sandpaper/issues/646#issuecomment-2829578435 + # only trigger checks manually if the validate-token step had no valid AWS or SANDPAPER_WORKFLOW token + - name: "Trigger checks" + if: | + steps.cpr.outputs.pull-request-number != '' && + steps.validate-org-workflow.outputs.is_valid != 'true' + run: | + gh workflow run docker_pr_receive.yaml --field pr_number=${{ steps.cpr.outputs.pull-request-number }} + shell: bash diff --git a/.github/workflows/update-workflows.yaml b/.github/workflows/update-workflows.yaml index 6414cf28..09ec1b63 100755 --- a/.github/workflows/update-workflows.yaml +++ b/.github/workflows/update-workflows.yaml @@ -1,55 +1,104 @@ -name: "02 Maintain: Update Workflow Files" - +name: "04 Maintain: Update Workflow Files" +description: "Update workflow files from the carpentries/sandpaper repository" on: + schedule: + - cron: '0 0 * * 2' workflow_dispatch: inputs: name: description: 'Who triggered this build (enter github username to tag yourself)?' required: true default: 'weekly run' + tarball: + description: 'Absolute URL to the desired sandpaper repo tarball' + required: false + default: '' clean: description: 'Workflow files/file extensions to clean (no wildcards, enter "" for none)' required: false default: '.yaml' - schedule: - # Run every Tuesday - - cron: '0 0 * * 2' jobs: - check_token: - name: "Check SANDPAPER_WORKFLOW token" - runs-on: ubuntu-22.04 - outputs: - workflow: ${{ steps.validate.outputs.wf }} - repo: ${{ steps.validate.outputs.repo }} - steps: - - name: "validate token" - id: validate - uses: carpentries/actions/check-valid-credentials@main - with: - token: ${{ secrets.SANDPAPER_WORKFLOW }} - update_workflow: name: "Update Workflow" - runs-on: ubuntu-22.04 - needs: check_token - if: ${{ needs.check_token.outputs.workflow == 'true' }} + runs-on: ubuntu-latest + permissions: + contents: write + pull-requests: write + id-token: write steps: - name: "Checkout Repository" uses: actions/checkout@v4 + - name: "Validate Current Org and Workflow" + id: validate-org-workflow + uses: carpentries/actions/validate-org-workflow@main + with: + repo: ${{ github.repository }} + workflow: ${{ github.workflow }} + + - name: Configure AWS credentials via OIDC + env: + role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }} + aws-region: ${{ secrets.AWS_GH_OIDC_REGION }} + if: | + steps.validate-org-workflow.outputs.is_valid == 'true' && + env.role-to-assume != '' && + env.aws-region != '' + uses: aws-actions/configure-aws-credentials@v5.0.0 + with: + role-to-assume: ${{ env.role-to-assume }} + aws-region: ${{ env.aws-region }} + + - name: Set PAT from AWS Secrets Manager + id: set-pat + env: + role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }} + aws-region: ${{ secrets.AWS_GH_OIDC_REGION }} + if: | + steps.validate-org-workflow.outputs.is_valid == 'true' && + env.role-to-assume != '' && + env.aws-region != '' + run: | + SECRET=$(aws secretsmanager get-secret-value \ + --secret-id carpentries-bot/github-pat \ + --query SecretString --output text) + PAT=$(echo "$SECRET" | jq -r .[]) + echo "::add-mask::$PAT" + echo "pat=$PAT" >> "$GITHUB_OUTPUT" + shell: bash + + - name: "Validate token" + id: validate-token + uses: carpentries/actions/check-valid-credentials@main + with: + token: ${{ steps.set-pat.outputs.pat || secrets.SANDPAPER_WORKFLOW }} + + - name: "No Token Found: Skipping Workflow Update" + if: ${{ steps.validate-token.outputs.wf == 'false' }} + run: | + echo "❗No valid SANDPAPER_WORKFLOW token or PAT from AWS found, cannot update workflows." + + echo "## ❌ Workflow Update Failed" >> $GITHUB_STEP_SUMMARY + echo "No valid SANDPAPER_WORKFLOW token or PAT from AWS found, cannot update workflows." >> $GITHUB_STEP_SUMMARY + shell: bash + - name: Update Workflows id: update + if: ${{ steps.validate-token.outputs.wf == 'true' }} uses: carpentries/actions/update-workflows@main with: - clean: ${{ github.event.inputs.clean }} + repo: ${{ github.event.inputs.tarball || 'https://carpentries.r-universe.dev' }} + clean: ${{ github.event.inputs.clean || '.yaml' }} - name: Create Pull Request id: cpr - if: "${{ steps.update.outputs.new }}" + if: | + steps.update.outputs.new && + steps.validate-token.outputs.wf == 'true' uses: carpentries/create-pull-request@main with: - token: ${{ secrets.SANDPAPER_WORKFLOW }} + token: ${{ steps.set-pat.outputs.pat || secrets.SANDPAPER_WORKFLOW }} delete-branch: true branch: "update/workflows" commit-message: "[actions] update sandpaper workflow to version ${{ steps.update.outputs.new }}" @@ -62,5 +111,5 @@ jobs: - Auto-generated by [create-pull-request][1] on ${{ steps.update.outputs.date }} [1]: https://github.com/carpentries/create-pull-request/tree/main - labels: "type: template and tools" + labels: "type: workflows" draft: false