Skip to content

Add pkg-precompile.jl script #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 61 commits into from
Apr 9, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
1d7e140
Designing tests
omus Apr 2, 2025
7f727ab
Improving tests
omus Apr 2, 2025
17ef5c8
Add more tests
omus Apr 3, 2025
a5da306
More stdlib tests
omus Apr 3, 2025
b13f82d
More tests
omus Apr 4, 2025
fa3d5cf
Some Julia 1.10 support
omus Apr 4, 2025
1bad93b
fixup! More tests
omus Apr 4, 2025
69170e0
Improving Julia 1.10 support
omus Apr 4, 2025
c13bacb
Julia 1.10 changes for same pkg version test
omus Apr 4, 2025
8565988
Tests working on Julia 1.10 and 1.11
omus Apr 4, 2025
ec3b96a
Improve bundled test
omus Apr 4, 2025
c0f5aa3
Add workflow
omus Apr 4, 2025
5ed93cf
fixup! Add workflow
omus Apr 4, 2025
89cd162
fixup! Add workflow
omus Apr 4, 2025
577e4ee
fixup! Add workflow
omus Apr 4, 2025
f86c140
fixup! Add workflow
omus Apr 4, 2025
666f777
fixup! Add workflow
omus Apr 4, 2025
ad9c40f
fixup! Add workflow
omus Apr 4, 2025
2d239f9
Drop unneeded import
omus Apr 4, 2025
04ad700
Permissions
omus Apr 4, 2025
bed42af
More versions
omus Apr 4, 2025
3ef5839
Fix 1.12
omus Apr 4, 2025
06b2b0e
fixup! Fix 1.12
omus Apr 4, 2025
aa472d8
Delete test/Manifest.toml
omus Apr 4, 2025
03a4cea
Skip 1.12 for now
omus Apr 4, 2025
df0cfd2
Fix test
omus Apr 4, 2025
164922c
Test all patch releases
omus Apr 4, 2025
a1d667c
Experiment with partial package
omus Apr 5, 2025
bbf07a9
Simple partial project fix
omus Apr 5, 2025
2564169
Revert "Simple partial project fix"
omus Apr 7, 2025
8a0e022
Support pre-existing compiled depot directories
omus Apr 7, 2025
48c7004
fixup! Support pre-existing compiled depot directories
omus Apr 7, 2025
b0e8ceb
Expect full precompile
omus Apr 7, 2025
1b290af
Handle root
omus Apr 7, 2025
3a0bdc0
Support named projects
omus Apr 7, 2025
6d4025c
Support partially transferred named projects
omus Apr 7, 2025
b43042a
Add re-instantiate test
omus Apr 7, 2025
a098c3e
Test workaround for re-instantiate
omus Apr 7, 2025
0fd7156
Remove duplicate testset
omus Apr 7, 2025
45bc4af
Add generate source placeholder
omus Apr 8, 2025
9d3102b
Add more comments
omus Apr 8, 2025
6eb62f9
Parallel test
omus Apr 8, 2025
0b0b85e
Add concurrency test
omus Apr 8, 2025
a27a5ce
Documentation and cleanup
omus Apr 8, 2025
959f1a1
fixup! Add concurrency test
omus Apr 8, 2025
b162ce8
Add README
omus Apr 8, 2025
c9f5879
Create gen-pkg-src.jl
omus Apr 8, 2025
fc3a9e6
fixup! Create gen-pkg-src.jl
omus Apr 8, 2025
a5a7cf5
fixup! Add concurrency test
omus Apr 8, 2025
ae48682
Show concurrent build output
omus Apr 8, 2025
2c02318
Iterating
omus Apr 8, 2025
868dbeb
fixup! Iterating
omus Apr 8, 2025
bd4ba4e
Update test
omus Apr 8, 2025
7a025a2
fixup! Update test
omus Apr 8, 2025
55054ee
Skip unnamed projects
omus Apr 8, 2025
475f846
Test Julia 1.12
omus Apr 8, 2025
ed0fd56
Remove duplicate statement
omus Apr 9, 2025
0518e35
Typo
omus Apr 9, 2025
9e9b265
Set fixed modification time in Julia
omus Apr 9, 2025
f2684a5
Stick to using separate fixed modification timestamp fix
omus Apr 9, 2025
98964ab
fixup! Stick to using separate fixed modification timestamp fix
omus Apr 9, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions .github/workflows/CI.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
---
name: CI
on:
pull_request:
paths:
- "pkg-precompile.jl"
- "test/**"
- ".github/workflows/CI.yaml"
push:
branches:
- main
tags: ["*"]

jobs:
version:
name: Resolve Julia Versions
# These permissions are needed to:
# - Checkout the Git repository (`contents: read`)
permissions:
contents: read
runs-on: ubuntu-latest
outputs:
json: ${{ steps.julia-version.outputs.resolved-json }}
steps:
- uses: actions/checkout@v4 # Needed for "min" to access the Project.toml
- uses: julia-actions/[email protected]
id: julia-version
with:
versions: |
- min # Oldest supported version
- lts # Long-Term Stable
- 1.10.0 # Earliest 1.10 release
- 1.10 # Latest 1.10 release
- 1.11.0 # Earliest 1.11 release
- 1.11 # Latest 1.11 release
- 1.12-nightly
project: test
if-missing: error

test:
name: Julia ${{ matrix.version }} - ${{ matrix.os }}
needs: version
# These permissions are needed to:
# - Delete old caches: https://github.com/julia-actions/cache#usage
permissions:
actions: write
contents: read
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
version: ${{ fromJSON(needs.version.outputs.json) }}
os:
- ubuntu-24.04
steps:
- uses: actions/checkout@v4
- uses: julia-actions/setup-julia@v2
with:
version: ${{ matrix.version }}
- uses: julia-actions/cache@v1
- name: Instantiate
shell: julia --color=yes --project=test {0}
run: |
using Pkg
Pkg.instantiate()
- name: Test
run: julia --color=yes --project=test test/runtests.jl
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
test/Manifest.toml
test/*/Manifest.toml
57 changes: 56 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1 +1,56 @@
# julia-container-scripts
# julia-container-scripts
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe julia-container-build-scripts or julia-docker-build-scripts is a better name?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is clear enough as far as I'm concerned.


Provides useful scripts for creating Julia container images.

## `pkg-precompile.jl`

The `pkg-precompile.jl` script supports creating Julia precompilation cache (`.ji`) files in a [build container cache mount](https://docs.docker.com/reference/dockerfile/#run---mounttypecache). By utilizing the build cache mount we can reuse precompilation files between builds which signficantly improves reduces Docker build times. A complete `Dockerfile` example can be seen below:

```Dockerfile
ARG JULIA_VERSION=1.11.4
FROM julia:${JULIA_VERSION}-bookworm AS julia-base

# Disable automatic package precompilation. We'll control when packages are precompiled.
ENV JULIA_PKG_PRECOMPILE_AUTO="0"

# Add registries required for instantiation
RUN julia --color=yes -e 'using Pkg; Pkg.Registry.add("General")'

# Limit Docker layer invalidation by only copying the Project.toml/Manifest.toml files.
ENV JULIA_PROJECT="/project"
COPY Project.toml *Manifest.toml ${JULIA_PROJECT}/

# TODO: Delete this optional statement if your Project.toml does not include the field
# "name" or you don't care about supporting the Julia versions listed below.
#
# Julia 1.10.0 - 1.10.6 and 1.11.0 require this source file to be present when
# instantiating a named Julia project.
Comment on lines +26 to +27
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1.10.8-1.10 require this for precompilation too.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I determined the versions which required this from using the test suite. We are testing for this in with the GEN_PKG_SRC Docker ARG which is only enabled on these Julia versions where it is required.

RUN curl -fsSLO https://raw.githubusercontent.com/beacon-biosignals/julia-container-scripts/refs/tags/v1/gen-pkg-src.jl && \
chmod +x gen-pkg-src.jl && \
./gen-pkg-src.jl && \
rm gen-pkg-src.jl

# Instantiate the Julia project environment and avoid precompiling. Ensure we perform a
# registry update here as changes to the Project.toml/Manifest.toml do not invalidate the
# Docker layer which added the registry.
RUN julia --color=yes -e 'using Pkg; Pkg.Registry.update(); Pkg.instantiate(); Pkg.build()'

# TODO: Delete this optional statement if you don't care about Julia 1.10 support or combine
# this statement with instantiate above to avoid bloating image size.
#
# Use a fixed modification time for all files in "packages" to avoid unnecessary precompile
# cache invalidation on Julia 1.10.
RUN julia -e 'VERSION < v"1.11" || exit(1)' && \
find "$(julia -e 'println(DEPOT_PATH[1])')/packages" -exec touch -m -t 197001010000 {} \;

# Precompile project dependencies using a Docker cache mount which persists between builds.
RUN --mount=type=cache,id=julia-depot,sharing=shared,target=/mnt/julia-depot \
curl -fsSLO https://raw.githubusercontent.com/beacon-biosignals/julia-container-scripts/refs/tags/v1/pkg-precompile.jl &&
chmod +x pkg-precompile.jl && \
./pkg-precompile.jl "/mnt/julia-depot" && \
rm pkg-precompile.jl

# Copy files necessary to load package and perform the first initialization.
COPY src ${JULIA_PROJECT}/src
RUN julia -e 'using Pkg; name = Pkg.Types.EnvCache().project.name; Pkg.precompile(name; timing=true); Base.require(Main, Symbol(name))'
```
26 changes: 26 additions & 0 deletions gen-pkg-src.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
#!/usr/bin/env -S julia --color=yes

using Pkg

env = Pkg.Types.EnvCache()
project_file = env.project_file
project_name = env.project.name

if project_name !== nothing
entry_file = joinpath(dirname(project_file), "src", "$(project_name).jl")

if !isfile(entry_file)
mkdir(dirname(entry_file))

# Define an empty module to avoid warnings such as:
# ```
# WARNING: --output requested, but no modules defined during run
# ```
open(entry_file, "w") do io
println(io, "module $(project_name)")
println(io, "end")
end
end
else
@warn "Julia project is unnamed and does not require an entry source file"
end
Loading