Migrate bots to GitHub Actions

[timvandermeij]

PDF.js has two dedicated bots for testing. However, this is all custom and maintained manually. This is already a problem now since the bots run quite old OS versions that lack a (modern enough) Python 3 version, which blocks the ttx upgrade for example (see #11802).

We could unblock this work by upgrading the OS versions, but for the longer term it would be better for the project to move towards a managed solution such as GitHub Actions. The latter integrates well with GitHub and our existing jobs, and reading quickly through the docs might allow us to create a botio-like flow where whitelisted users can comment on PRs to trigger actions (builds).

In terms of our builds, I mainly see a challenge for the browser tests since they are stateful, so we need to be able to fetch/store the reference images somewhere. I'm thinking this could be done in an S3 bucket, and let GitHub manage the credentials for that.

Let's investigate if these solutions could work for us and how much logic we have to maintain ourselves. The objective is to use existing solutions as much as possible so we don't have to build/maintain it ourselves, as well as improving reliability.

[brendandahl]

For the long term health of the project I think it would be good to migrate all testing to github actions. As Tim mentioned we can store refs in S3 or alternatively Marco mentioned gh action artifacts can be stored for 90 days so we could set up a cron job to maintain them.

[timvandermeij]

I think using GitHub artifacts might also be a good and perhaps even easier solution since everything then stays within the GitHub ecosystem, so we don't have to manage another AWS account for S3. The cronjob should be doable.

[timvandermeij]

Time permitting I'll try to set up a basic workflow to run the bot commands using GitHub Actions, without artifact retention for now, so we can see how/if it works and go from there.

[timvandermeij]

I have been working on migrating the preview task from the bot to GitHub Actions in timvandermeij@fe60c41 and while mostly working, I did run into the limitation that build artifact files can't seem to be accessed individually, at least through the UI, since the are all combined into one ZIP file. There are actions to post a comment on a PR with a link to the artifact, but if we can't get a direct link that opens the viewer in the browser that'll be inconvenient. Something similar will be needed for the reftest analyzer for migrating the reference testing task.

This limitation is already reported in actions/upload-artifact#14, so given that GitHub doesn't really "host" the artifacts separately, we might still need to use an S3 bucket to make the preview/reftest analyzer available in the browser from a direct link, similar to how Botio does it. We can then also store the reference images there, avoiding the need for a cronjob.

@brendandahl Would it be possible for you to set up an S3 bucket and configure the required secrets for accessing it in GitHub's secret manager? Once that's done, we can test uploading our artifacts and reference images there. See https://dev.to/johnkevinlosito/deploy-static-website-to-s3-using-github-actions-4a0e for how to do this.

[timvandermeij]

I now have a working implementation of Botio's lint command in a GitHub Actions workflow in commit timvandermeij@d37aa7d. It works the same as Botio: if a specific comment is placed on a pull request by an authorized user, it will trigger the command and put notification comments on the pull request. timvandermeij#5 (comment) is an example of how that looks (first a commit that failed linting, then one that made it pass).

I chose the lint command since it's easiest to set up the issue comment infrastructure with since we don't have to bother with running browsers. However, while Botio has the linting command, I'm not sure if we should also introduce that in the new setup given that linting by default already runs in the CI workflow for every push, so I don't think there is ever a point where we want to run linting on-demand since it already happens (and if it fails, we can simply retrigger the CI build). Therefore, once I get the browsers working, we can start introducing workflows for all Botio tasks that don't require assets since that's pending on the S3 point above (basically everything except for the reference tests).

I'm glad that we now know that we can have the same behavior as Botio with the workflows. Overall this migration will make the build infrastructure nicely managed and easier to work with than the current bots. Issues like #11802 will be easier to address too because the build infrastructure is managed centrally and a simple pull request will be enough to update it, so we don't need direct machine access anymore.

[Snuffleupagus]

One thing that has always annoyed me a little bit with the botio setup is the need to have separate bot-specific tasks for all the tests we want to run (look at the gulp-tasks whose names start with "bot...").

In the new "GitHub Actions" setup, would it be possible to use command line flags instead together with the "normal" gulp-tasks (to reduce duplication)?
For example, instead of having to run gulp bottest, could we run gulp test --bot and have createTestSource directly check for that flag?

[timvandermeij]

I don't see why not. In fact, I'm not really sure why that's even needed in the current Botio setup given that I don't see anything stopping us from using gulp test --bot at https://github.com/mozilla/botio-files-pdfjs/blob/master/on_cmd_test.js#L42? Botio or GitHub Actions will simply run the command we provide it, so if the Gulpfile can handle that argument we're good.

[brendandahl]

@brendandahl Would it be possible for you to set up an S3 bucket and configure the required secrets for accessing it in GitHub's secret manager? Once that's done, we can test uploading our artifacts and reference images there. See https://dev.to/johnkevinlosito/deploy-static-website-to-s3-using-github-actions-4a0e for how to do this.

Yeah, we should be able to do that. Let me look into getting it set up.

[brendandahl]

@timvandermeij I've added an s3 bucket. I've also added the following repository secrets:

AWS_ACCESS_KEY_ID
AWS_REGION
AWS_S3_BUCKET
AWS_SECRET_ACCESS_KEY

[timvandermeij]

Nice; thank you! It will require some more tweaking of my initial GitHub Actions workflows, but I think we now have all the pieces of the puzzle to implement all features that our current bots have.

[calixteman]

Will we have access (using ssh or whatever) to the machines where the actions are ran ?
I ask because few times I had to hack directly on them.

[timvandermeij]

Not that I know of. GitHub Actions' workers get destroyed after the job is done, just like at Travis CI and other CIs. We can save artifacts in the S3 bucket if need be though.