Minimum image

[toastie89]

My backup strategy highly relies on an older version of this fabulous container image. Over time many extra packages, such as curl, bash, bash completion, apprise or even clients for different databases have been included.

From a security standpoint I would feel much better if the container would only contain a minimum set of binaries, especially as it is running as root and has access to all my sensitive data. Further each extra layer adds complexity and possible failure which I would definitely want to avoid when it comes to backup.

I would love to see a minimum image which can be build upon. What do you think about?

[grantbevis]

Good idea, something we will look to do soon

[toastie89]

I've tried to figure out the required packages below. Does anyone know why curl and ca-certificates are there?

[r] required
[o] optional
[?] unclear

Dockerfile -----------------------
  [r] tzdata            -- required for correct time stamps
  [o] sshfs             -- optional, only when using ssh remote repositories
  [?] openssl           -- ?! borg dependency for encryption
  [r] fuse              -- required for backup restore
  [?] ca-certificates   -- ?!
  [o] logrotate         -- optional, used when logs are not handled by docker #112
  [r] lz4-libs          -- required, borg dependency
  [r] libacl            -- required, borg dependency
  [o] postgresql-client -- optional, only required for backup DBs
  [o] mariadb-client    -- optional, only required for backup DBs
  [o] mongodb-tools     -- optional, only required for backup DBs
  [?] curl              -- #?! #60
  [o] findmnt           -- optional, to check if mount exists #94
  [o] bash              -- optional, #117
  [o] bash-completion   -- optional, #117
  [o] bash-doc          -- optional, #117

requirements.txt -----------------------
  [o] apprise           -- optional, only required for push notification
  [r] borgbackup        -- borg
  [r] llfuse            -- required for backup restore
  [r] borgmatic         -- borgmatic

[modem7]

@toastie89 - I can't see any reason ca-certs is required at all currently (but might be useful in scenarios like #154). Might have been a legacy requirement for when we built the wheels during image creation.

I've just done a build without it and it seems to build fine (haven't tested the container, but I see no reason it should fail).

The other thing we can do with a simple base image, is just build the feature-full image from the base image, reducing build time of any extensions.

E.g. FROM ghcr.io/borgmatic-collective/borgmatic:base

[bb]

When implementing this, please try not to add yet another flavor folder to this repository. I'm speaking specifically about the git repository, not the published Docker images. I'd very much prefer having the compose file(s), a single .env.template and a single README at the to level of the repository.

Proposal

This is to improve the current state, maybe as a first step before splitting a minimal version.

docker-compose.ntfy.yml

version: '3'
services:
  borgmatic:
    image: b3vis/borgmatic:${FLAVOR-ntfy}
    volumes:
      - ${VOLUME_NTFY-./data/nfty}:/root/.config/ntfy   

.env.template

# you can only chose *one* of the flavors
# COMPOSE_FILE=docker-compose.yml:docker-compose.ntfy.yml # uncomment to use NTFY
# COMPOSE_FILE=docker-compose.yml:docker-compose.msmtp.yml # uncomment to use msmtp
# COMPOSE_FILE=docker-compose.yml:docker-compose.everything.yml # with coreutils, jq, whatever 😉 

[...]

# for NTFY flavour only, uncomment if you don't want to use the default
# VOLUME_NTFY=./data/ntfy

As a side effect, using default values for the ENV variables would also simplify adding features like VOLUME_BORGMATIC_STATE -- this change was needed in 3 compose files, but with the approach outlined above only in one. The referenced commit misses amending the three .env.template files which could been made superfluous by using a default value.

Having only one template would also prevent mistakes like me adding the hint in the wrong env.template long time ago. 🙄

[modem7]

When implementing this, please try not to add yet another flavor folder to this repository.

We should have gotten rid of ntfy and msmtp a while ago as they've long been replaced by apprise and are both redundant.

I'd imagine there would be two images.

• Base
• Extended

Scaling an env template for both minimal and standard would be easy to get into a single template as standard would just be an extension on top of base.

Eg.

#.env
#base/minimal
1=1
2=2
3=3

#extended
4=4
5=5
6=6

[grantbevis]

The way I see this going down is as @modem7 said, base and extended images. Base has the bare essentials and extended will have Apprise and other goodies in it. This is possibly something I will complete under the supercronic switch on #186

[bb]

You're preaching to the converted/choir. I simply didn't know yet that ntfy/msmtp were deprecated. I didn't check the closed issues when coming back to this project after more than a year. Maybe this should be reflected in the Readme.

As said, I'm talking about the repository layout. From the comment above, just replace docker-compose.ntfy.yml with docker-compose.extended.yml:

version: '3'
services:
  borgmatic:
    image: b3vis/borgmatic:extended
    ... volumes etc. for extended

And my request still stands: can we have both base and extended at the top level of the repository, please? I don't want to see two folders base and an extended, at least on the level of Readme, docker-compose.* an .env.template files.

[grantbevis]

Oh yes @bb - I agree the sub directories are annoying. Keep an eye on the PR #186 and I will simplify this over the Christmas break

[bb]

I hope you had a great Christmas break @grantbevis. I'm still postponing my upgrade from the mid-2021 version I'm currently running. Do you have an estimate on your big refactoring? I'd rather start over after the refactoring and not directly before it.

[grantbevis]

@bb - Apologies I missed your previous comment. I have now merged my supercronic changes and the new image includes apprise let me know if you have any questions

[bb]

Thanks for coming back to this.
No questions for usage so far.

I guess next steps for the repo are something like this:

git rm msmtp
git rm ntfy
git mv base/* .

Should I prepare a PR or do you want to go in a different direction?

[grantbevis]

I will take a proper look and have a think @bb and work out how best to approach this. I want to keep it simple and have the base.Dockerfile contain only the bare essentials required as per the comments previously. I do however feel the base image should contain the mariaDB/PostgreSQL tools for backing up databases as this is a core feature of borgmatic.

[grantbevis]

P.S I wish to flatten the repo too so the base/ directory will be gone

[toastie89]

I would prefer to have a rock solid base image to build upon without heavy db clients. Further when I use the image for PostgreSQL, I would want to install a client from the same release and not expect the client to be included.

[grantbevis]

I would prefer to have a rock solid base image to build upon without heavy db clients. Further when I use the image for PostgreSQL, I would want to install a client from the same release and not expect the client to be included.

So we could achieve that with a EXTRA_PKGS envar that the entry.sh will install during first run or would you want pre-baked images with different db clients baked in?

[kaechele]

Where are we at with this? I have a repo going locally where I've been playing around with this.

I was wondering whether the default latest tag should be the fully featured image and there would be a minimal or base tag for the people that have this kind of minimalism requirement. My rationale there is that it's probably a better out-of-the-box user experience to have an image that doesn't require extra steps to use the most popular features while still allowing more advanced users to select a minimal image or use it to build their own images off of.

For the sake of simplicity and maintainability I would also only supply two flavours: minimal and latest (which is the fully featured image).
For anything in between I assume people would have the ability to build their own version off of the base image if they had this requirement.

[modem7]

Where are we at with this?

This will probably be looked at once we look at removing the other images and unify things a bit more.

@grantbevis and I have a timeline plan, but it'll be a couple of months before we do the rework as it'll affect most of the project including documentation etc, so we'll need to sit down and create a plan.

[kaechele]

Sounds good! Thanks!
Let us know if there's anything we can help with.

[modem7]

Hey guys,

I've just raised #324 which means we can start work on this.

There are two ways I can see this going:

1. latest is a minimal image, and S6 can install additional packages later (I can see this biting us on the ass, especially in slow/no internet locations).
2. Two tags, two images, additional maintenance, especially if we start mucking around with S6 in any fundamental way.
3. Keep image as it is, use S6 to remove packages during bootup. This is arguably the easiest option, but not really a "minimal image".

Any other ideas? Option 2 is probably the sanest option, but does increase resource costs in terms of maintenance.

[toastie89]

An earlier fork of this project saved my files for years, and I'm grateful for the great work! Following the development, I've realized that this project and Borgmatic become too sophisticated for my use case. While I've initially planned to contribute a minimal version and started this issue, I've ended up running a cron-scheduled Borg in a container with some script to push status and stats to Prometheus. The work is available here, in case someone wants to make use of it.

[modem7]

@toastie89 - Shall we close this ticket for now, and if you ever want to submit a minimal version, then we can revisit it?

[toastie89]

Sounds like a plan.