Skip to content

Commit e7aa389

Browse files
committed
docker: Simplify Docker builds
- Changed: Drop support for Archlinux. This wasn't being used and was not guaranteed to work. Instead, it is better to target the latest Ubuntu versions, even if they are not LTS releases. - Changed: Move Docker files out of `dist/docker`. We only have one now. And this way the Makefile can be better re-used by optional packages. - Changed: Docker images no longer tagged based on options. This was a big headache to use, especially when other images based on these. - Changed: Use stage-based builds. This is better for debugging. - Changed: Running smoketests no longer part of build. It's important that the test whether the image works is performed in the domain in which it will be used. This is not during build time. In addition, this can allow certain build improvements. - Changed: Main Makefile no longer has any Docker targets. It's better to expose users and developers to the Makefile.docker, which is more orthogonal and easier to understand what will happen. - Fixed: Caching behavior required changes to Ubuntu default configuration.
1 parent aede5d9 commit e7aa389

13 files changed

+264
-290
lines changed

.dockerignore

+12-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,16 @@
1+
# Git data not needed:
12
.git/
3+
4+
# Docker recipe not required in image:
5+
Dockerfile
6+
Makefile.docker
7+
8+
# Secrets should be mounted with --secret:
9+
setup.sh
10+
11+
# Build artifacts:
212
build/
313
**/build/
4-
dist/docker/
14+
15+
# Optional packages are built separately:
516
optional/

.gitignore

+3
Original file line numberDiff line numberDiff line change
@@ -36,3 +36,6 @@ callgrind.out.*
3636
# Miscellaneous
3737
*.log
3838
/.gtm/
39+
40+
# Docker configuration
41+
setup.sh

dist/docker/Dockerfile.ubuntu renamed to Dockerfile

+25-18
Original file line numberDiff line numberDiff line change
@@ -6,24 +6,30 @@
66
#
77
# If you are behind a proxy, make sure to pass in the respective HTTP_PROXY,
88
# HTTPS_PROXY, and NO_PROXY variables.
9+
#
10+
# Note to maintainer:
11+
# Make sure you repeat any ARG required after every FROM statement.
12+
ARG UBUNTU_NAME=ubuntu
913
ARG UBUNTU_VERSION=20.04
10-
FROM ubuntu:${UBUNTU_VERSION}
14+
ARG UBUNTU_IMAGE=${UBUNTU_NAME}:${UBUNTU_VERSION}
15+
FROM ${UBUNTU_IMAGE} AS stage-setup-system
16+
ARG UBUNTU_VERSION
1117

1218
# Install System Packages
1319
#
1420
# These packages are required for building and testing Cloe.
1521
COPY Makefile.help /cloe/Makefile.help
1622
COPY Makefile.setup /cloe/Makefile.setup
17-
RUN --mount=type=cache,id=ubuntu-${UBUNTU_VERSION}-cache,target=/var/cache/apt \
18-
--mount=type=cache,id=ubuntu-${UBUNTU_VERSION}-lib,target=/var/lib/apt \
23+
RUN rm -f /etc/apt/apt.conf.d/docker-clean; echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache
24+
RUN --mount=type=cache,id=ubuntu-${UBUNTU_VERSION}-cache,target=/var/cache/apt,sharing=locked \
25+
--mount=type=cache,id=ubuntu-${UBUNTU_VERSION}-lib,target=/var/lib/apt,sharing=locked \
1926
apt-get update && \
2027
apt-get install -y make ccache locales libbsd0 && \
2128
make -f /cloe/Makefile.setup \
2229
DEBIAN_FRONTEND=noninteractive \
2330
APT_ARGS="--no-install-recommends -y" \
2431
install-system-deps && \
25-
locale-gen && \
26-
rm -rf /var/lib/apt/lists/*
32+
locale-gen
2733

2834
ENV LANG=C.UTF-8
2935
ENV LC_ALL=C.UTF-8
@@ -37,16 +43,18 @@ RUN pip3 install --upgrade pip && \
3743

3844
# Install and Setup Conan
3945
#
40-
# You may not want to use the default Conan remote (conan-center), so we use
46+
# You may not want to use the default Conan remote (conancenter), so we use
4147
# whatever is stored in the build arguments CONAN_REMOTE.
4248
#
4349
# If you need to login to a Conan remote, make use of the setup.sh file, which
4450
# will be sourced for every run argument that uses the conan command.
4551
#
4652
# The following profiles are available: default, cloe-release, cloe-debug
47-
COPY dist/conan /cloe/dist/conan
53+
FROM stage-setup-system AS stage-setup-conan
4854
ARG CONAN_PROFILE=cloe-release
4955
ENV CONAN_NON_INTERACTIVE=yes
56+
57+
COPY dist/conan /cloe/dist/conan
5058
RUN make -f /cloe/Makefile.setup setup-conan && \
5159
conan config set general.default_profile=${CONAN_PROFILE} && \
5260
conan profile update options.cloe-engine:server=False ${CONAN_PROFILE}
@@ -56,14 +64,16 @@ RUN make -f /cloe/Makefile.setup setup-conan && \
5664
# All common processes are made easy to apply by writing target recipes in the
5765
# Makefile at the root of the repository. This also acts as a form of
5866
# documentation.
59-
WORKDIR /cloe
67+
FROM stage-setup-conan AS stage-vendor
68+
ARG VENDOR_TARGET="export-vendor download-vendor"
6069
ARG KEEP_SOURCES=0
6170

71+
WORKDIR /cloe
72+
6273
# Download or build dependencies:
6374
COPY vendor /cloe/vendor
6475
COPY Makefile.package /cloe
6576
COPY Makefile.all /cloe
66-
ARG VENDOR_TARGET="export-vendor download-vendor"
6777
RUN --mount=type=cache,target=/ccache \
6878
--mount=type=secret,target=/root/setup.sh,id=setup,mode=0400 \
6979
if [ -r /root/setup.sh ]; then . /root/setup.sh; fi && \
@@ -77,10 +87,13 @@ RUN --mount=type=cache,target=/ccache \
7787
conan remove \* -b -f; \
7888
fi
7989

80-
# Build Cloe.
81-
COPY . /cloe
90+
# Build Cloe:
91+
FROM stage-vendor AS stage-build
8292
ARG PROJECT_VERSION=unknown
83-
ARG PACKAGE_TARGET="package smoketest-deps"
93+
ARG PACKAGE_TARGET="export smoketest-deps"
94+
ARG KEEP_SOURCES=0
95+
96+
COPY . /cloe
8497
RUN --mount=type=cache,target=/ccache \
8598
--mount=type=secret,target=/root/setup.sh,id=setup,mode=0400 \
8699
if [ -r /root/setup.sh ]; then . /root/setup.sh; fi && \
@@ -94,9 +107,3 @@ RUN --mount=type=cache,target=/ccache \
94107
else \
95108
conan remove \* -b -f; \
96109
fi
97-
98-
# Run smoketests.
99-
RUN --mount=type=secret,target=/root/setup.sh,id=setup,mode=0400 \
100-
if [ -r /root/setup.sh ]; then . /root/setup.sh; fi && \
101-
make smoketest && \
102-
conan user --clean

Makefile

+1-17
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ help::
3737
include Makefile.setup
3838

3939
# Workspace targets -----------------------------------------------------------
40-
.PHONY: lockfile status deploy sphinx doxygen docker-all docker-test docker-release purge-all export-cli smoketest smoketest-deps
40+
.PHONY: lockfile status deploy sphinx doxygen purge-all export-cli smoketest smoketest-deps
4141
help::
4242
$(call print_help_section, "Available workspace targets")
4343
$(call print_help_target, status, "show status of each of the Conan packages")
@@ -49,10 +49,6 @@ help::
4949
$(call print_help_target, deploy-cli, "install ${_yel}cloe-launch${_rst} with ${_dim}${PIPX}${_rst}")
5050
$(call print_help_target, export-cli, "export ${_yel}cloe-launch-profile${_rst} Conan recipe")
5151
echo
52-
$(call print_help_target, docker-test, "build only a single Docker image")
53-
$(call print_help_target, docker-all, "build all Docker images")
54-
$(call print_help_target, docker-release, "upload all Conan packages from Docker images")
55-
echo
5652

5753
${BUILD_LOCKFILE}:
5854
${MAKE} -f Makefile.package SOURCE_CONANFILE=/dev/null LOCKFILE_SOURCE=${LOCKFILE_SOURCE} ${BUILD_LOCKFILE}
@@ -78,18 +74,6 @@ deploy-cli:
7874
export-cli:
7975
${MAKE} -C cli conan-profile
8076

81-
docker-test:
82-
$(call print_header, "Building ubuntu-18.04 Docker image...")
83-
${MAKE} -C dist/docker ubuntu-18.04
84-
85-
docker-all:
86-
$(call print_header, "Building all Docker images...")
87-
${MAKE} -C dist/docker all
88-
89-
docker-release:
90-
$(call print_header, "Uploading all Conan packages from Docker images...")
91-
${MAKE} -C dist/docker release
92-
9377
sphinx:
9478
$(call print_header, "Generating Sphinx documentation...")
9579
mkdir -p ${BUILD_DIR}/sphinx

Makefile.docker

+150
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
# Makefile.docker
2+
#
3+
# This file defines all Docker targets.
4+
5+
PROJECT_ROOT := $(dir $(abspath $(lastword ${MAKEFILE_LIST})))
6+
PROJECT_VERSION := $(shell make --no-print-directory -C ${PROJECT_ROOT} -f Makefile.package info-version)
7+
8+
UBUNTU_NAME := ubuntu
9+
UBUNTU_VERSIONS := 18.04 20.04 22.04
10+
11+
DOCKER := DOCKER_BUILDKIT=1 docker
12+
DOCKER_IMAGE_NAME := cloe/cloe-engine
13+
DOCKER_IMAGE_VERSION := ${PROJECT_VERSION}
14+
DOCKER_IMAGE := ${DOCKER_IMAGE_NAME}:${DOCKER_IMAGE_VERSION}
15+
DOCKER_CONTEXT := ${PROJECT_ROOT}
16+
17+
DOCKER_USER_ARGS +=
18+
DOCKER_BUILD_ARGS += --build-arg PROJECT_VERSION=${PROJECT_VERSION}
19+
DOCKER_RUN_ARGS += -it --rm
20+
21+
ifndef (${https_proxy},)
22+
DOCKER_NETWORK := host
23+
DOCKER_BUILD_ARGS += --network=host \
24+
--build-arg https_proxy="${https_proxy}" \
25+
--build-arg http_proxy="${http_proxy}" \
26+
--build-arg no_proxy="${no_proxy}"
27+
DOCKER_RUN_ARGS += --network=host \
28+
--env https_proxy="${https_proxy}" \
29+
--env http_proxy="${http_proxy}" \
30+
--env no_proxy="${no_proxy}"
31+
else
32+
DOCKER_NETWORK := internal
33+
endif
34+
35+
ifeq ($(shell [ -f ${PROJECT_ROOT}/setup.sh ] && echo "true"),true)
36+
DOCKER_BUILD_ARGS += --secret id=setup,src=${PROJECT_ROOT}/setup.sh
37+
DOCKER_RUN_ARGS += -v ${PROJECT_ROOT}/setup.sh:/root/setup.sh
38+
endif
39+
40+
ifneq (${CONAN_PROFILE},)
41+
DOCKER_BUILD_ARGS += --build-arg CONAN_PROFILE=${CONAN_PROFILE}
42+
endif
43+
44+
ifneq (${VENDOR_TARGET},)
45+
DOCKER_BUILD_ARGS += --build-arg VENDOR_TARGET=${VENDOR_TARGET}
46+
endif
47+
48+
ifneq (${PACKAGE_TARGET},)
49+
DOCKER_BUILD_ARGS += --build-arg PACKAGE_TARGET=${PACKAGE_TARGET}
50+
endif
51+
52+
ifeq (${KEEP_SOURCES},1)
53+
DOCKER_BUILD_ARGS += --build-arg KEEP_SOURCES=1
54+
endif
55+
56+
# -----------------------------------------------------------------------------
57+
58+
include ${PROJECT_ROOT}/Makefile.help
59+
60+
.PHONY: help
61+
.SILENT: help
62+
help::
63+
$(call print_help_usage)
64+
echo
65+
$(call print_help_section, "Available Docker targets")
66+
$(call print_help_target, all, "build and test all Ubuntu versions")
67+
$(call print_help_target, ubuntu-VERSION, "build and test the Ubuntu VERSION image")
68+
$(call print_help_target, build-all, "build all Ubuntu versions")
69+
$(call print_help_target, build-ubuntu-VERSION, "build the Ubuntu VERSION image")
70+
$(call print_help_target, test-all, "build all Ubuntu versions")
71+
$(call print_help_target, test-ubuntu-VERSION, "test the Ubuntu VERSION image")
72+
$(call print_help_target, run-ubuntu-VERSION, "run the Ubuntu VERSION image")
73+
$(call print_help_target, release-all, "release all Ubuntu versions")
74+
$(call print_help_target, release-ubuntu-VERSION, "release the Ubuntu VERSION image")
75+
$(call print_help_target, remove-current-images, "remove and prune all ${DOCKER_IMAGE} Docker images")
76+
$(call print_help_target, remove-all-images, "remove and prune all ${DOCKER_IMAGE_NAME} Docker images")
77+
echo
78+
$(call print_help_section, "User configuration")
79+
$(call print_help_define, CONAN_PROFILE, ${CONAN_PROFILE})
80+
$(call print_help_define, VENDOR_TARGET, ${VENDOR_TARGET})
81+
$(call print_help_define, PACKAGE_TARGET, ${PACKAGE_TARGET})
82+
$(call print_help_define, KEEP_SOURCES, ${KEEP_SOURCES})
83+
$(call print_help_args_lines, DOCKER_USER_ARGS, ${DOCKER_USER_ARGS})
84+
echo
85+
$(call print_help_section, "Docker configuration")
86+
$(call print_help_define, UBUNTU_NAME, "${UBUNTU_NAME}")
87+
$(call print_help_define_lines, UBUNTU_VERSIONS, ${UBUNTU_VERSIONS})
88+
$(call print_help_define, DOCKER, "${DOCKER}")
89+
$(call print_help_define, DOCKER_CONTEXT, "${DOCKER_CONTEXT}")
90+
$(call print_help_define, DOCKER_NETWORK, "${DOCKER_NETWORK}")
91+
$(call print_help_define, DOCKER_IMAGE, "${DOCKER_IMAGE}")
92+
$(call print_help_args_lines, DOCKER_BUILD_ARGS, ${DOCKER_BUILD_ARGS})
93+
$(call print_help_args_lines, DOCKER_RUN_ARGS, ${DOCKER_RUN_ARGS})
94+
echo
95+
96+
.SILENT: FORCE
97+
FORCE:
98+
# Targets containing patterns (%) cannot be made .PHONY. The standard
99+
# solution to work around this is to add a FORCE target to the
100+
# prerequisites of every target with patterns.
101+
102+
.PHONY: build-all
103+
build-all: $(addprefix build-ubuntu-,${UBUNTU_VERSIONS})
104+
105+
.PHONY: test-all
106+
test-all: $(addprefix test-ubuntu-,${UBUNTU_VERSIONS})
107+
108+
.PHONY: release-all
109+
release-all: $(addprefix release-ubuntu-,${UBUNTU_VERSIONS})
110+
111+
ubuntu-%: FORCE build-ubuntu-% test-ubuntu-%
112+
@echo
113+
@echo "Completed building and testing: ${DOCKER_IMAGE}-$@"
114+
115+
.PHONY: all
116+
all: $(addprefix ubuntu-,${UBUNTU_VERSIONS})
117+
118+
build-ubuntu-%: FORCE Dockerfile
119+
${DOCKER} build -f Dockerfile ${DOCKER_BUILD_ARGS} ${DOCKER_USER_ARGS} \
120+
--build-arg UBUNTU_VERSION=$* \
121+
-t ${DOCKER_IMAGE}-ubuntu-$* \
122+
${DOCKER_CONTEXT}
123+
124+
test-ubuntu-%: FORCE
125+
docker run ${DOCKER_RUN_ARGS} ${DOCKER_USER_ARGS} ${DOCKER_IMAGE}-ubuntu-$* \
126+
bash -ec "[ -f /root/setup.sh ] && source /root/setup.sh; make smoketest"
127+
128+
run-ubuntu-%: FORCE
129+
docker run ${DOCKER_RUN_ARGS} ${DOCKER_USER_ARGS} ${DOCKER_IMAGE}-ubuntu-$*
130+
131+
release-ubuntu-%: FORCE
132+
@test -f setup.sh || echo 'Error: require setup.sh for user authentication'
133+
${DOCKER} run ${DOCKER_RUN_ARGS} ${DOCKER_USER_ARGS} ${DOCKER_IMAGE}-ubuntu-$* \
134+
bash -ec 'source /root/setup.sh && conan upload --force --all -c "*"'
135+
136+
.PHONY: require-setup-sh
137+
require-setup-sh:
138+
@if [ ! -f ${PROJECT_ROOT}/setup.sh ]; then \
139+
echo "ERROR: require ${PROJECT_ROOT}/setup.sh to proceed"; \
140+
exit 1; \
141+
fi
142+
143+
.PHONY: remove-current-images
144+
remove-current-images:
145+
docker rmi $$(docker images --format '{{.Repository}}:{{.Tag}}' | grep '${DOCKER_IMAGE}')
146+
147+
.PHONY: remove-all-images
148+
remove-all-images:
149+
docker rmi $$(docker images --format '{{.Repository}}:{{.Tag}}' | grep '${DOCKER_IMAGE_NAME}:')
150+

README.md

+66-1
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@ all packages instead of re-using packages it finds:
101101
```
102102
Run `make help` to get an overview of the available targets we expect you to
103103
use. For more details on how this is done, have a look at the Makefiles in the
104-
repository root or the Dockerfiles in `dist/docker` directory.
104+
repository root.
105105
106106
If you experience timeout issues waiting for Conan Center, the reason is likely
107107
the boost dependency's hundreds of binary packages. You can then slightly
@@ -183,6 +183,71 @@ this shouldn't need to be performed manually.
183183
184184
See the documentation on testing Cloe for more details.
185185
186+
### Building Docker Images
187+
188+
The `Dockerfile` provided in the repository root can be used to create one
189+
or more Docker images. These can be used as devcontainers or simply to
190+
create Conan packages in a reproducible environment.
191+
192+
The `Makefile.docker` contains multiple targets to build, test, and
193+
release Docker images for the current Cloe version in the repository.
194+
Run `make -f Makefile.docker help` to get an impression of what is possible:
195+
196+
$ make -f Makefile.docker help
197+
Usage: make target
198+
199+
Available Docker targets:
200+
all build and test all Ubuntu versions
201+
ubuntu-VERSION build and test the Ubuntu VERSION image
202+
build-all build all Ubuntu versions
203+
build-ubuntu-VERSION build the Ubuntu VERSION image
204+
test-all build all Ubuntu versions
205+
test-ubuntu-VERSION test the Ubuntu VERSION image
206+
run-ubuntu-VERSION run the Ubuntu VERSION image
207+
release-all release all Ubuntu versions
208+
release-ubuntu-VERSION release the Ubuntu VERSION image
209+
remove-current-images remove and prune all cloe/cloe-engine:0.19.0 Docker images
210+
remove-all-images remove and prune all cloe/cloe-engine Docker images
211+
212+
User configuration:
213+
CONAN_PROFILE=
214+
VENDOR_TARGET=
215+
PACKAGE_TARGET=
216+
KEEP_SOURCES=
217+
DOCKER_USER_ARGS=
218+
219+
Docker configuration:
220+
UBUNTU_NAME=ubuntu
221+
UBUNTU_VERSIONS=\
222+
18.04
223+
20.04
224+
22.04
225+
DOCKER=DOCKER_BUILDKIT=1 docker
226+
DOCKER_CONTEXT=/home/captain/cloe
227+
DOCKER_NETWORK=host
228+
DOCKER_IMAGE=cloe/cloe-engine:0.19.0
229+
DOCKER_BUILD_ARGS=\
230+
--build-arg PROJECT_VERSION=0.19.0
231+
DOCKER_RUN_ARGS=\
232+
-it
233+
--rm
234+
235+
Because Docker images may be built in environments that have a proxy running,
236+
the Makefile will automatically add the proxy variables if they are are
237+
detected in the host environment.
238+
239+
The section `User configuration` shows variables that can be set to modify
240+
the resulting Docker image; if they are empty here, then the defaults in
241+
the `Dockerfile` are used, which should suffice for most use-cases.
242+
243+
If you want to use a different Conan remote from the default, you need to
244+
copy `setup.sh.example` to `setup.sh` and modify the values to match your
245+
environment.
246+
247+
Note that this build requires the use of docker buildx, which has been
248+
available for some time. This allows us to mount secrets in at build time
249+
and also speeds up the build by the strategic use of caches.
250+
186251
[1]: https://conan.io
187252
[2]: https://docs.microsoft.com/en-us/windows/wsl/about
188253
[3]: https://ubuntu.com

0 commit comments

Comments
 (0)