Skip to content

Running out of space: Share same _work directory for all runners? #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Akronae opened this issue Nov 28, 2024 · 3 comments
Closed

Running out of space: Share same _work directory for all runners? #21

Akronae opened this issue Nov 28, 2024 · 3 comments
Assignees

Comments

@Akronae
Copy link

Akronae commented Nov 28, 2024

Hi,

Thank you for your tool, it's really handy helping us manage ~30 runners.
One big issue I'm facing is that my 85GB SSD machines are quickly running out of storage.

This is mainly caused by the infamous node_modules folder being replicated for all runners (up to 8 on each machine).
It would be really helpful if those _work directories could be shared among runners.
In essence it could not only save space but also time I guess.

Do you think it could be doable?
I'd be happy to make a PR.

image
image
image

@vbem
Copy link
Owner

vbem commented Dec 1, 2024

Hi Akronae,

I am very pleased to receive your feedback on usage.

Actually, the _work directory is the home directory of many workflow contexts, such as ${{ runner.temp }}, ${{ runner.workspace }}, ${{ github.workspace }}, etc. Sharing it among multi runners on the same host may cause conflicts of different workflow runs.

However, if the node_modules folder is the largest duplication occupant, maybe you can simply create a shared folder on the Linux host (e.g. /home/SharedFolder), and set the NODE_PATH: /home/SharedFolder/.node_modules environment variable in all related workflow files. Please note that as all runners created by this application were already in a shared Linux group runners, the /home/SharedFolder should also be owned by this group.

The disk usage may also increase because logs, workspace caches, runners updates, etc. You can also re-create the runners in crontab daily or weekly as a more complete housekeeping.

@vbem vbem closed this as completed Dec 1, 2024
@Akronae
Copy link
Author

Akronae commented Dec 2, 2024

Ok, thanks for your help, I chose the recreate all runners periodically as this will also fix the issue where half the runners disconnect and go offline after a month or so.

This seems to work fine:

#!/bin/bash
for i in {0..7}
do
  bash multi-runners/mr.bash del --user runner-$i
  bash multi-runners/mr.bash add --org <my org>
done

Cheers 😄

@vbem
Copy link
Owner

vbem commented Dec 2, 2024

Awesome. Besides, starting from v1.2.0, del --org <your-org-name> --count 8 and add --org <your-org-name> --count 8 were also supported.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants