multi-runners

🌈🌈🌈 Multi self-hosted GitHub action runners on single host! 🌈🌈🌈

Introduction

This application is designed for controlling multi self-hosted GitHub Action runners on single host, when Actions Runner Controller (ARC) is not feasible in your engineering environment. This application has following advantages:

• Single Linux host required.
• Single Bash script.
• Lightweight wrapper of GitHub official self-hosted runner.
• Both github.com and GitHub Enterprise are support.
• Either organization or repository or GitHub Cloud Enterprise level runners are supported.

Usage

mr.bash - https://github.com/vbem/multi-runners

Environment variables:
  MR_GITHUB_BASEURL=https://github.com
  MR_GITHUB_API_BASEURL=https://api.github.com
  MR_RELEASE_URL=<latest on github.com/actions/runner/releases>
  MR_USER_BASE=<default in /etc/default/useradd>
  MR_USER_PREFIX=runner-
  MR_GITHUB_PAT=ghp_***

Sub-commands:
  add       Add one self-hosted runner on this host
            e.g. ./mr.bash add --org ORG --repo REPO --labels cloud:ali,region:cn-shanghai
            e.g. ./mr.bash add --org ORG --count 3
  del       Delete one self-hosted runner on this host
            e.g. ./mr.bash del --user runner-1
            e.g. ./mr.bash del --org ORG --count 3
  list      List all runners on this host
            e.g. ./mr.bash list
  download  Download GitHub Actions Runner release tar to /tmp/
            Detect latest on github.com/actions/runner/releases if MR_RELEASE_URL empty
            e.g. ./mr.bash download
  pat2token Get runner registration token from GitHub PAT (MR_GITHUB_PAT)
            e.g. ./mr.bash pat2token --org SOME_OWNER --repo SOME_REPO

Options:
  --enterprise  GitHub Cloud Enterprise name, optional
  --org         GitHub organization name
  --repo        GitHub repository name, registration on organization-level if empty
  --user        Linux local username of runner
  --labels      Extra labels for the runner
  --group       Runner group for the runner
  --token       Runner registration token, takes precedence over MR_GITHUB_PAT
  --dotenv      The lines to set in runner's '.env' files
  --count       The number to add or del, optional, defaults to 1 for add and all for del
  --opts        Extra options for 'config.sh', optional, such as '--no-default-labels'
  -h --help     Show this help.

Download this application

This application requires to be run under a Linux user with non-password sudo permission (e.g., %runners ALL=(ALL) NOPASSWD:ALL). It's also fine to run this application by root:

git clone https://github.com/vbem/multi-runners.git
cd multi-runners
./mr.bash --help

Setup PAT

This application requires a GitHub personal access token with smallest permissions and shortest expiration time. Only add/del/pat2token sub-commands need this PAT. You can remove it on GitHub after multi-runners' setup.

PAT types	Repository level runners	Organization level runners
Fine-grained PAT (recommended)	Referring to repository API, the administration:write permission is required.	Referring to organization policy & organization API, the organization_self_hosted_runners:write permission is required.
Classic PAT	Referring to repository API, need the repo scope	Refer to organization API, need the admin:org scope; if the repository is private, repo scope is also required.

During runtime, you can set your PAT in environment variable MR_GITHUB_PAT. To simplify subsequent execution, you can define any environment variable in .env file. For example,

# .env file under the directory of this application
MR_GITHUB_PAT='github_pat_***********'
ALL_PROXY=socks5h://localhost

You can run following command to check whether or not your PAT can generate GitHub Actions runners' registration-token:

./mr.bash pat2token --org <ORG-NAME> --repo <REPO-NAME>

Download the latest version of GitHub Actions package

If environment variable MR_RELEASE_URL is empty, this application will download the latest version of GitHub Actions runners tar package to local directory /tmp/ during runtime.

./mr.bash download

If your Linux host is internet bandwidth limited, you can also manually upload it from laptop to /tmp/<tar.gz file name>, and set the MR_RELEASE_URL env in .env file, e.g. /tmp/actions-runner-linux-x64-2.345.6.tar.gz.

GitHub Enterprise Server editions

GitHub Enterprise Server editions usually have different server and API URL prefixes comparing with github.com, you can set them in environment variables MR_GITHUB_BASEURL and MR_GITHUB_API_BASEURL.

GitHub Enterprise Cloud level registration

For GitHub Enterprise Cloud level registration, you can specify the --enterprise option to set the GitHub Enterprise Cloud name.

Setup multi-runners on single host

To setup multi-runners, you can simplify run following command multi times:

# 1 runner for repository `<ORG-NAME-1>/<REPO-NAME-1>`
./mr.bash add --org <ORG-NAME-1> --repo <REPO-NAME-1>

# 2 runners for repository `<ORG-NAME-1>/<REPO-NAME-2>`
./mr.bash add --org <ORG-NAME-1> --repo <REPO-NAME-2> --count 2

# 3 runners for organization `<ORG-NAME-2>`
./mr.bash add --org <ORG-NAME-2> --count 3

This application will create one Linux local user for one runner via useradd command. The Base Directory of these users is read from HOME setting in your /etc/default/useradd file by default (typically /home). You can also set it in environment variable MR_USER_BASE to override system-wide default.

List all runners on current host

This application also integrated status check of runners.

./mr.bash list

Which outputs,

runner-0 537M running https://github.com/<ORG-NAME-1>/<REPO-NAME-1>
runner-1 537M running https://github.com/<ORG-NAME-1>/<REPO-NAME-2>
runner-2 537M running https://github.com/<ORG-NAME-1>/<REPO-NAME-2>
runner-3 537M running https://github.com/<ORG-NAME-2>
runner-4 537M running https://github.com/<ORG-NAME-2>
runner-5 537M running https://github.com/<ORG-NAME-2>

Delete an existing runner

# delete an existing runner by its local Linux username.
./mr.bash del --user <runner-?>

# delete all runners for specific repository
./mr.bash del --org <ORG-NAME-1> --repo <REPO-NAME-2>

# delete multi runners by `--count` options.
./mr.bash del --org <ORG-NAME-2> --count 2

Specify runner in workflow file

In jobs.<job_id>.runs-on, target runners can be based on the labels as follows via GitHub context:

# For organization level self-hosted runners
runs-on: [self-hosted, '${{ github.repository_owner }}']

# For repository level self-hosted runners
runs-on: [self-hosted, '${{ github.repository }}']

Set environment variables into runners process

As described in GitHub official document, there's an approach to inject environment variables into runners process via the .env file before configuring or starting the self-hosted runners. This can be achieved via the --dotenv option, for example:

./mr.bash add --org <ORG> --repo <REPO> --dotenv 'TZ=Asia/Shanghai' --dotenv 'PATH=\$PATH:/mybin' --dotenv 'all_proxy=socks5h://localhost:1080'

Then the following lines will be added to .env file located in self-hosted runner's directory before its configuring and starting:

TZ=Asia/Shanghai
PATH=$PATH:/mybin
all_proxy=socks5h://localhost:1080

Inject hook script before starting the runner service

This application also supports to inject a hook script before starting the runner service via MR_CMD_SVC_PRE_START environment variable.

For example, you can limit the CPU and RAM usage of the runner service by modifying the unit file of the runner service. Add the following lines to the .env file under the directory of this application. Then this script will be executed just before starting the runner service - ./svc.sh start.

MR_CMD_SVC_PRE_START="$(cat <<-'__HEREDOC__'
echo "🚀 Running custom commands before starting runner service."
name="$(<mr.d/name)"
echo "🚀 Runner: $name | User: $USER | Dir: $PWD"
source <( sed -n "1,$(grep -n '^UNIT_PATH=' svc.sh | cut -d: -f1)p" svc.sh )
echo "🚀 UNIT_PATH: $UNIT_PATH"
sudo sed -i -e '/^\[Service\]/a CPUQuota=50%' -e '/^\[Service\]/a MemoryMax=512M' "$UNIT_PATH"
sudo systemctl daemon-reload
echo "🚀 Updated unit file:"
cat "$UNIT_PATH"
__HEREDOC__
)"