Skip to content

[Windows] Remove engine intermediate engine build products before committing ue4-minimal builder image #265

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

slonopotamus
Copy link
Collaborator

These products are not used after build step, but they take massive amounts of disk space: up to 350GBs for multi-platform (Win64+PS4+PS5+XB1+XSX) engine build

To make matters even worse, current Docker image commit algorithm on Windows additionally uses x2 of resulting image size on disk.

…mitting ue4-minimal builder image

These products are not used after build step,
but they take massive amounts of disk space:
up to 350GBs for multi-platform (Win64+PS4+PS5+XB1+XSX) engine build

To make matters even worse,
current Docker image commit algorithm on Windows additionally uses x2 of resulting image size on disk.
@TBBle
Copy link
Collaborator

TBBle commented Jun 21, 2022

That's pretty nuclear since it is blowing away stuff that was already there from the ue4-source image, but I guess we don't care about that directory after that point, and deletions will produce a much smaller container image that additions.

It's a shame there's no way (AFAIK) to tell the builder "Ignore changes under this directory" when capturing a RUN result.

Does the Linux pipeline need/want the same improvement?

@slonopotamus
Copy link
Collaborator Author

That's pretty nuclear since it is blowing away stuff that was already there from the ue4-source image

I don't have an easier way to wipe away build stuff. We already removed .git, so we can't git clean -fdx. Even if we could, there are GitDependencies-downloaded files that we can't easily separate from build products.

Does the Linux pipeline need/want the same improvement?

On Linux, there are several points:

  1. Appropriate Dockerfile RUN directive is not committed at all, thanks to BuildKit
  2. There are some COPY steps that need to be adjusted somehow.

@slonopotamus
Copy link
Collaborator Author

UPD: this change doesn't save size of VHDX itself (because VHDX doesn't shrink when you remove files), but it does save 2x of build products size during image commit. And obviously, image commit is much faster since it doesn't need to write all those GBs to disk.

@slonopotamus slonopotamus merged commit b36516a into adamrehn:master Jun 21, 2022
@slonopotamus slonopotamus deleted the windows-disk-usage-optimization branch June 21, 2022 21:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants