Add INCLUDE to Dockerfile parsing in bud

[TomSweeneyRedHat]

Description
Cut from #850 by @pixdrift - I think this should be a stand alone discussion/enhancement request.

I also wanted to raise in this thread the possibility of inclusion of an INCLUDE feature in the Dockerfile parser (bud), to essentially achieve the same as above. This has been raised / rejected several times in Docker/moby issues, but interested to know if there is any precedent in Buildah for support of additional features in the Dockerfile, or if bud is purely for compatibility with existing features.

Proposal discussions for INCLUDE in Docker/Moby:
moby/moby#735
moby/moby#12749

I don't really see this as including a entire existing Dockerfiles (ie. with their own FROM), more to include common code blocks/snippets. I have achieved the same using native buildah run and source to pull in common code blocks, but would still be interested in comments on the above idea.

[vrothberg]

I've been thinking about such a feature for a while and as you like building stuff on old unix tools, you might be interested in using the C pre-processor. It does everything you need which is mainly textual replacement via the #include directive but it could also allow you to do macro expansion and even conditional rules and way more things.

Consider the following (minimal) example for demonstration:

Dockerfile-base
FROM alpine                                                          
RUN  echo "very important and complex common patterns" > /pattern.txt

Dockerfile-derived
#include "Dockerfile-base"
RUN echo "my own stuff" > /own-stuff.txt

Now run $ cpp -E Dockerfile-derived > Dockerfile-final. We could do all kinds of things with that and pre-process each Dockerfile by default.

[pixdrift]

@vrothberg, thanks for the comment / suggestion, it's nice as it does have the relationship described inside the file.

With this suggestion, I am trying to move away from processing/parsing outside of the Dockerfilers as I want everything described in the Dockerfile and for users to be able to run a single buildah bud step with no dependant steps/scripts so I can maintain their simplicity of use.

It's interesting because you are including in the opposite direction to what I am suggesting, as the bud processor will want the FROM in the initial image. With your example i'd use a FROM to Dockerfile-base... but then you end up storing the dependant image relationships for rebuilding etc. I appreciate this was just an example though and may not be the only use case 😃

I have also seen a make file used to construct the final image but my issue there is that the Dockerfile you have executed isn't what's in your version control (because it's constructed/built in the pipeline), so the final product of the make ends up being another artefact to store.

This is what I am hoping for.

Dockerfile-myapp:
FROM centos                                                     
INCLUDE Dockerfile-snippet-patching-update
RUN <install myapp steps etc>
INCLUDE Dockerfile-snippet-common-entry

Dockerfile-snippet-patching-update:
COPY repo.conf /etc/yum/repos.d/
RUN yum clean all && yum update -y && rm -rf /var/cache/yum
RUN <more patch specific stuff eg. packages outside of yum/rpm>

Dockerfile-snippet-common-entry:
COPY entrypoint.sh /
ENV <common env stuff>
ENTRYPOINT /entrypoint.sh

The following is what I am using currently (as you may have seen), which does similar to the include I describe (using just bud), but requires description of the relationship in a script outside of the Dockerfiles:
#850

I'm not quite in utopia yet! 😁

-edit-

Something that's worth noting is that ideally you would reference a Dockerfile directory for the INCLUDE, so each INCLUDE could have self contained external elements (eg. files) that it uses in its ADD/COPY commands.

[vrothberg]

@pixdrift Thanks for the nice explanation!

Please don't get me wrong, I am not against extending Dockerfile to support an INCLUDE directive. I think it's something urgently needed and many users do such things already but have to work around that.

The major concern I have is that adding such a directive to Buildah would make those Dockerfiles incompatible to Moby/Docker and all other Dockerfile-based tools. When using cpp instead, those files could still be used for other tools after being pre-processed. In case of buildah-bud this could happen transparently within Buildah by unconditionally pre-processing each input Dockerfile before parsing.

But I think it depends on what you want to achieve. If the plan is to provide some proof of concept to Docker/Moby, then the supporting INCLUDE is the way to go (at the cost of breaking compatibility with other tools). If the plan is to generally support composing images on a finer granularity than images (i.e., FROM), then I would recommend to not diverge from other tools. But maybe, we don't care about the other tools at that point and want to empower users to finally use this feature?

[pixdrift]

@vrothberg, this is almost exactly the question I put forward to the others working on buildah. Is the role of bud to implement Dockerfile compatibility, and everything else should happen in the 'native' buildah commands, so perhaps buildah include? Or is there an opportunity to extend the functionality for buildah specific Dockerfiles (Buildahfiles?) that maybe would have some adoption from others? (but obviously causes a fracture in 'standards'). It's really a bigger question than just the INCLUDE directive detailed here, but definitely want to include this in the discussion!

I'm basically chasing a solution that meets the following requirements/constraints.
1. The file/files that is stored in version control can be executed directly by the build tool (ie. no intermediate interpretation step to 'create' the Dockerfile) as this is an additional artefact to store.
2. No logic/configuration needs to exist outside the files to describe their relationship
3. Access to all existing capabilities that can be used in existing Dockerfiles from the segments/snippets (ie. not just RUN commands)
4. Segments/snippets can be inserted at arbitrary locations in the file, not just concatenated to the end / in a certain order (see my above example, an item is included, then some tasks, then another include)
5. Segments/snippets can be referenced outside the Dockerfile directory context so they can be shared by multiple Dockerfiles

The options I have seen/worked with so far:

buildah bud -f

bud -f in its old behaviour was very close, but it fails item (2.) and (4.) above. The order in which the files are 'spliced' is codified in the command, and they are concatenated in order, essentially achieving the same as cat on the command line, so arbitrary insertion into a 'parent' file isn't really possible so the test case doesn't work for the above example.

Example of this form:

buildah bud -f Dockerfile -f ../Dockerfile-snippet-patching-update -f ../Dockerfile-snippet-common-entry Dockerfile-myapp

Repository of scripts (eg .sh files)

Adding for example a tarball of scripts and then executing them selectively. This is OK, but when using Dockerfile syntax, with RUN, your snippets then can't include other Dockerfile commands such as ENTRYPOINT, ENV etc (so they are missing in the below example). This method fails (3.)

Example of this form: (WARNING: below example is untested/theoretical)

[Dockerfile-myapp]

FROM centos
ADD allmyscripts.tar.gz /scripts                                                
RUN /scripts/Dockerfile-snippet-patching-update
RUN <install myapp steps etc>
RUN /scripts/Dockerfile-snippet-common-entry

make or cpp (ie. external build tools)

@vrothberg suggestion above. Very close to complete solution, I have seen make used in the past, my only concern is that it does need the interpretation/parsing by another tool (1.). I realise it's low overhead, but i'm considering the (minor) impact to end user experience using multiple tools.

Example of this form: (WARNING: below example is untested/theoretical)

[Dockerfile-myapp]

FROM centos                                                     
#include "Dockerfile-snippet-patching-update"
RUN <install myapp steps etc>
#include "Dockerfile-snippet-common-entry"

[Dockerfile-snippet-patching-update]

COPY repo.conf /etc/yum/repos.d/
RUN yum clean all && yum update -y && rm -rf /var/cache/yum
RUN <more patch specific stuff eg. packages outside of yum/rpm>

[Dockerfile-snippet-common-entry]

COPY entrypoint.sh /
ENV <common env stuff>
ENTRYPOINT /entrypoint.sh

Compile step of:

$ cpp -E Dockerfile-myapp > Dockerfile-final

buildah native using 'source'

Basically using buildah in the form of a bash file, and using source to pull in other bash script segments/snippets. I had a PoC of this up and running, but because you're at the shell, I find the resulting execution isn't as 'contained' as it is in the bud interpreter which can lead to bad things (ie. bad escapes in RUN commands being executed on host if not done correctly). You also need to maintain consistency in variable names etc. to maintain context.

Example of this form: (WARNING: below example is untested/theoretical)

[Dockerfile-myapp.sh]

#!/bin/bash
newcontainer=$(buildah from centos)
scratchmnt=$(buildah mount ${newcontainer})
source Dockerfile-snippet-patching-update
buildah run  ${newcontainer} <install myapp steps etc>
source Dockerfile-snippet-common-entry

[Dockerfile-snippet-patching-update.sh]

cp repo.conf ${scratchmnt}/etc/yum/repos.d/
buildah run ${newcontainer} yum clean all
buildah run ${newcontainer} yum update -y
rm -rf ${scratchmnt}/var/cache/yum
buildah run ${newcontainer} <more patch specific stuff eg. packages outside of yum/rpm>

[Dockerfile-snippet-common-entry.sh]

cp entrypoint.sh ${scratchmnt}/
buildah config --env ${newcontainer} <common env stuff>
buildah config --entrypoint ${newcontainer} /entrypoint.sh

[vrothberg]

I'm basically chasing a solution that meets the following requirements/constraints.

1. The file/files that is stored in version control can be executed directly by the build tool (ie. no intermediate interpretation step to 'create' the Dockerfile) as this is an additional artefact to store.
2. No logic/configuration needs to exist outside the files to describe their relationship>
3. Access to all existing capabilities that can be used in existing Dockerfiles from the segments/snippets (ie. not just RUN commands)
4. Segments/snippets can be inserted at arbitrary locations in the file, not just concatenated to the end / in a certain order (see my above example, an item is included, then some tasks, then another include)
5. Segments/snippets can be referenced outside the Dockerfile directory context so they can be shared by multiple Dockerfiles

All of this can be done with cpp. Note that $ cpp -E Dockerfile-myapp > Dockerfile-final is not required and can be done transparently in buildah-bud. But I don't want to hijack the discussion regarding cpp. As you stated, it's important to first have an agreement on the general issue whether buidah can/should diverge from other Dockerfile-based tools or not.

[pixdrift]

The function/behaviour is correct with the cpp solution. I believe the implementation should fail on existing Docker interpreters so it doesn't get skipped with the leading #. Not sure how specific you were about interpreting the cpp as is in the example.

There is a kind of precedent in grep using the -E to enable extended capability, perhaps the implementation could be similar with bud?

Consider the following Dockerfiles

[container_app/Dockerfile]

FROM docker.io/library/centos:7
RUN echo pixdrift
E_INCLUDE container_inc

[container_inc/Dockerfile]

RUN echo inside.include

In this example I have prefixed the INCLUDE with E_ to protect it from a keyword collision if Docker do ever implement an INCLUDE capability, which would result in ambiguity. This I guess could be anything, eg. BUILDAH_INCLUDE.

I think the implementation should require the user to explicitly switch the bud parser into extended mode to enable the additional features, so there is no confusion with regards to compatibility (perhaps even warn users when this is enabled). For example with -e and --extended options to bud.

Some examples

buildah bud as it exists today (failure expected)

Failure expected because an extended feature has been enabled without enabling the extended capability.

# buildah bud container_app
STEP 1: FROM docker.io/library/centos:7
STEP 2: RUN echo pixdrift
pixdrift
STEP 3: E_INCLUDE container_inc
ERRO[0000] Build error: Unknown instruction: "e_include" 

This error could be enhanced to something like this:

ERRO[0000] Build error: Attempt to use extended instruction: "e_include" . Enable the extended capabilities with "buildah bud --extended"

buildah bud with 'extended' capability enabled (success expected)

# buildah bud -e container_app
STEP 1: FROM docker.io/library/centos:7
STEP 2: RUN echo pixdrift
pixdrift
STEP 3: E_INCLUDE container_inc
STEP 4: RUN echo inside.include
inside.include

If a FROM appears in container_inc/Dockerfile, a warning could be shown to the user so they are aware it's ignored, eg.

STEP 3: E_INCLUDE container_inc
WARN[0000] Build warning: Ignoring FROM statement in included Dockerfile

build attempt with moby/Docker existing implementation (failure expected)

Failure expected because the command is listed in the Dockerfile and is picked up by Docker.

# docker build container_app
Sending build context to Docker daemon 2.048 kB
Error response from daemon: Unknown instruction: E_INCLUDE

Not sure if it's also worth discussing implementation details of levels of E_INCLUDE to support? to avoid complex dependency structures. eg. E_INCLUDE can only occur in the Dockerfile directly referenced by buildah bud? (this may be an unnecessary constraint/limit).

-edit-

Was also going to mention the potential to detect the file name and change behaviour. eg. if it's Buildahfile, the extended capability could be automatically turned on for the bud operation.

[rhatdan]

5 years of discussion on adding this feature...
I kind of like the idea of adding cpp support and only add INCLUDE once it is fully available, but that still leads us to the point where Dockerfiles used with buildah can not be used with Docker.

Perhaps we should have a meeting to discuss this.

[vrothberg]

I kind of like the idea of adding cpp support and only add INCLUDE once it is fully available, but that still leads us to the point where Dockerfiles used with buildah can not be used with Docker.

That's true, but I think it's the lesser evil as those Dockerfiles can easily be transformed into something other parsers can eat.

Perhaps we should have a meeting to discuss this.

In case, I'd be interested in joining.

[pixdrift]

@rhatdan The issue I see with the cpp method is that it will fail silently in Docker, and result in a successful Docker build with missing segment/snippet.

I have proposed the E_INCLUDE method above so it's clear that it extends standard Dockerfile and breaks legacy compatibility if someone attempts to build the file that contains extensions with Docker.

[rhatdan]

@vbatts @nalind @mheon @smarterclayton @wking WDYT

[smarterclayton]

At that point it's not a Dockerfile. If there's a desire for other formats, it would be best to make it be clear that it's something else (via a separate command, or flags that explicitly identify it, or a different filename). Dockerfiles are valuable because they're reasonably portable and identical everywhere. Creating something new needs to be different (it's a new API in a new direction).

[pixdrift]

@smarterclayton, that is why I have proposed the '-e' flag above, which makes it clear that it isn't compatible with the Dockerfile standard. This makes sure that when users are enabling the extended features they know they are breaking compatibility.

I'm not sure it warrants a whole new file format / parser, even if it does break compatibility, but if it's a genuine concern the Buildahfile I proposed or even Podmanfile, for example, could be used with the same parser (and extended options automatically enabled for these filenames).