Add coreutils and jq pkgs

[psi-4ward]

Request: Add coreutils and jq to the container. It adds only about 2,16 MB.

Motivation: More flexibility when executing scripts (ie hooks) inside the container.

In my case I've written simple backup-space_exporter.sh and borg_exporter.sh which provide some Prometheus metrics.

Could also be useful when working with docker-api.

[modem7]

Going by #144 - I'm wondering if the image is getting a bit bloated with extra packages as is.

Might it be worthwhile just adding a new section to the entrypoint script for additional packages as a variable seeing as the Borgmatic container wouldn't be restarted all that often?

Something as simple as:

#!/bin/bash
# current entrypoint script
...
# /current entrypoint script

# Install extra packages if set
if [ -v EXTRA_PKGS ]
then
    echo -e "\nInstalling extra packages..."
    echo -e "Packages chosen: $EXTRA_PKGS\n"
    apk add -v $EXTRA_PKGS
    echo -e "Packages installed\n"
fi

With a compose file such as:

    environment:
      CRON: $BORG_CRON
      CRON_COMMAND: $BORG_CRON_COMMAND
      EXTRA_PKGS: "coreutils jq"

Will give us the following during container startup if the variable is met:

root@AlpineDev16293:/$ bash test.sh

start current entrypoint script...blah blah blah content

Installing extra packages...
Packages chosen: coreutils jq

fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
(1/7) Installing libacl (2.3.1-r1)
(2/7) Installing libattr (2.5.1-r1)
(3/7) Installing skalibs (2.12.0.1-r0)
(4/7) Installing utmps-libs (0.1.2.0-r1)
(5/7) Installing coreutils (9.1-r0)
(6/7) Installing oniguruma (6.9.8-r0)
(7/7) Installing jq (1.6-r2)
Executing busybox-1.35.0-r18.trigger
OK: 50 packages, 190 dirs, 2672 files, 36 MiB
Packages installed

rest of entrypoint script...blah blah blah content

Otherwise, if the variable is null (not set or empty) then we just get:

root@AlpineDev17051:/$ bash test.sh

start current entrypoint script...blah blah blah content

rest of entrypoint script...blah blah blah content

This way, any additional packages people wish to install to expand their experience using the container can be satisfied at container startup, without having to change how things are for everyone else + bloat the container for 7 million* people (* not actually 7 million, but just going by docker pulls), as well as allow for quick addition/removal of packages if people wish.

We're already doing a similar thing with the proposed Cron settings in #150 - I'm happy to add it either as part of or as a separate pull request on top of it once it gets merged?

[psi-4ward]

Yes (and no).

Trivial things first: apk add -v --no-cache $EXTRA_PKGS

I generally would be okay with this option but then the image is not fully idempotent (as long as you don't specify the exact version of the pkg). The next problem could be that often containers cant access the internet directly (only through HTTP-Proxy or even dont have any access).

So in my op: Dont implement this solution.

But ... we could think about init-scripts.

if [ -e /init.d ]; then
  for i in /init.d/*.sh ; do 
    /init.d/$i
  done
fi

With this hack ppl could do arbitrary stuff when the container starts

ie

apk add --no-cache jq coreutils
wget http://.../prom_exporter.sh

or

wget -o /etc/borgmatic.d/config.yaml http://[email protected]/.../borg_config.yaml

or ... whatever?

[modem7]

@psi-4ward Remember that Alpine works slightly different than other distributions such as Ubuntu where each OS release only contains a single "latest" version of a package, so it'd update whenever the base image is updated to the latest version of Alpine.

E.g. If I'm on python:alpine3.16:latest the available version of coreutils is coreutils-9.1-r0

If I try to install any other version, it will not work.

root@Alpine_PythonDev2521:/$ apk add coreutils=9.0-r2
ERROR: unable to select packages:
  coreutils-9.1-r0:
    breaks: world[coreutils=9.0-r2]

In regard to specialised environments such as via VPN, air gapped etc, things get more complicated and catering to all solutions is entering a world of hurt, especially with additional file dependencies etc, and as such, I would argue those requirements really should spin off a fork for their specific requirements.

There are all manners of ways of implementing highly complex customisations, waitforit scripts etc, but I would argue the scope of this project is relatively limited to Borgmatic and directly supporting packages, so a simple "if defined, then add, otherwise, don't" would probably be all that would be viable in our current scope.

The other issue is support. Once we start adding additional features as part of the main package, we're also having to support different configurations if they go wrong, and we're not a large team.

---

On another (non-Borgmatic) note, if you want a quick pre-setup Dev environment - I've got a project to spin up one quickly that you can call from something like Windows Terminal: https://github.com/modem7/docker-devenv (although if you just need basic things, then a simple docker run --rm -it alpine /bin/sh would work obviously)

[psi-4ward]

I would argue those requirements really should spin off a fork for their specific requirements.

agree, one who would use this should fork the image and very likely already has an internal registry.

---

So how to arrange with this? Probably including the borgmatic prometheus exporter script would be a nice feature for others and we close this issue and look if we bundle the exporter?

$EXTRA_PKGS is simple and could be added easily => OK for me

leave as it is would be also OK for me and I'll create fork

[toastie89]

What about just using the existing image as base to build from an individual Dockerfile?

When it comes to Prometheus, I have used grep 😉

#!/bin/sh

url='http:// example:9090/push/metrics/job/borgmatic/host/example'

json=`borgmatic info --last 1 --json`

cat <<EOF | curl --data-binary @- $url
# TYPE borgmatic_run_duration_seconds gauge
# HELP borgmatic_run_duration_seconds duration between start and end
`echo -n 'borgmatic_run_duration_seconds ' && \
echo $json | grep -o '"duration": [0-9]*' | grep -o '[0-9]*'`

# TYPE borgmatic_run_size_deduplicated_bytes gauge
# HELP borgmatic_run_size_deduplicated_bytes deduplicated size of th
e last run
`echo -n 'borgmatic_run_size_deduplicated_bytes ' && \
echo $json | grep -o '"compressed_size": [0-9]*' | grep -o '[0-9]*'`

# TYPE borgmatic_run_size_original_bytes gauge
# HELP borgmatic_run_size_original_bytes original size of the last r
un
`echo -n 'borgmatic_run_size_original_bytes ' && \
echo $json | grep -o '"original_size": [0-9]*' | grep -o '[0-9]*'`

# TYPE borgmatic_cache_chunks_size_total_compressed_bytes gauge
# HELP borgmatic_cache_chunks_size_total_compressed_bytes total size
 of all chunks
`echo -n 'borgmatic_cache_chunks_size_total_compressed_bytes ' && \
echo $json | grep -o '"unique_csize": [0-9]*' | grep -o '[0-9]*'`
EOF

[psi-4ward]

What about just using the existing image as base to build from an individual Dockerfile?

Yes possible but I thought adding only some few MBs would be okay ^^

---

#!/bin/bash
set -eu -o pipefail

# Default values for BORGMATIC_BIN and BORGMATIC_CONF env-vars
BORGMATIC_BIN=${BORGMATIC_BIN:-/usr/local/bin/borgmatic}
BORGMATIC_CONF=${BORGMATIC_CONF:-/etc/borgmatic.d/config.yml}

# We need jq to parse borg output
if ! command -v jq &> /dev/null; then
  >&2 echo "ERROR: jq executable could not be found."
  exit 1
fi
if ! command -v $BORGMATIC_BIN &> /dev/null; then
  >&2 echo "ERROR: $BORGMATIC_BIN executable could not be found."
  exit 1
fi

BORGMATIC="$BORGMATIC_BIN -nc -c $BORGMATIC_CONF"

BORG_LIST="$($BORGMATIC list --json)"

if [ $(echo "$BORG_LIST" | jq -r '. | length') -gt 1 ]; then
  # TODO: We could easily loop through all configured repos.
  >&2 echo "ERROR: This borgmatic exporter supports only one repository."
  exit 1
fi

REPO_ARCHIVES="$(echo "$BORG_LIST" | jq -r '.[0].archives | length')"
REPOSITORY=$(echo "$BORG_LIST" | jq -r '.[0].repository.location')
BACKUP_TARGET=$(echo ${REPOSITORY//ssh:\/\//} | cut -d/ -f1)

LABELS="{host=\"$(hostname)\",repository=\"${REPOSITORY}\",backup_target=\"${BACKUP_TARGET}\"}"

BORG_INFO=$($BORGMATIC info --last 1 --json)

REPO_SIZE=$(echo "$BORG_INFO" | jq -r '.[0].cache.stats.total_size')
REPO_COMPRESSED_SIZE=$(echo "$BORG_INFO" | jq -r '.[0].cache.stats.total_csize')
REPO_DEDUPLICATED_SIZE=$(echo "$BORG_INFO" | jq -r '.[0].cache.stats.unique_csize')
REPO_CHUNKS=$(echo "$BORG_INFO" | jq -r '.[0].cache.stats.total_chunks')
REPO_UNIQUE_CHUNKS=$(echo "$BORG_INFO" | jq -r '.[0].cache.stats.total_unique_chunks')

LAST_ARCHIVE_DATE=$(echo "$BORG_INFO" | jq -r '.[0].archives[0].end')
LAST_ARCHIVE_TIMESTAMP=$(date -u -d "$LAST_ARCHIVE_DATE" +%s)
LAST_ARCHIVE_FILES=$(echo "$BORG_INFO" | jq -r '.[0].archives[0].stats.nfiles')
LAST_ARCHIVE_ORIGINAL_SIZE=$(echo "$BORG_INFO" | jq -r '.[0].archives[0].stats.original_size')
LAST_ARCHIVE_DEDUPLICATED_SIZE=$(echo "$BORG_INFO" | jq -r '.[0].archives[0].stats.deduplicated_size')
LAST_ARCHIVE_COMPRESSED_SIZE=$(echo "$BORG_INFO" | jq -r '.[0].archives[0].stats.compressed_size')
LAST_ARCHIVE_DURATION=$(echo "$BORG_INFO" | jq -r '.[0].archives[0].duration' | cut -d. -f1)

echo "# HELP borg_repo_archives Number of archives in this repository"
echo "# TYPE borg_repo_archives gauge"
echo "borg_repo_archives${LABELS} ${REPO_ARCHIVES}"

echo "# HELP borg_repo_size_bytes Size of the repository in bytes"
echo "# TYPE borg_repo_size_bytes gauge"
echo "borg_repo_size_bytes${LABELS} ${REPO_SIZE}"

echo "# HELP borg_repo_compressed_size_bytes Compressed size of the repository in bytes"
echo "# TYPE borg_repo_compressed_size_bytes gauge"
echo "borg_repo_compressed_size_bytes${LABELS} ${REPO_COMPRESSED_SIZE}"

echo "# HELP borg_repo_deduplicated_size_bytes Deduplicated size of the repository in bytes"
echo "# TYPE borg_repo_deduplicated_size_bytes gauge"
echo "borg_repo_deduplicated_size_bytes${LABELS} ${REPO_DEDUPLICATED_SIZE}"

echo "# HELP borg_repo_chunks Total number of chunks in the repository"
echo "# TYPE borg_repo_chunks gauge"
echo "borg_repo_chunks${LABELS} ${REPO_CHUNKS}"

echo "# HELP borg_repo_unique_chunks Number of unique chunks in the repository"
echo "# TYPE borg_repo_unique_chunks gauge"
echo "borg_repo_unique_chunks${LABELS} ${REPO_UNIQUE_CHUNKS}"


echo "# HELP borg_last_archive_timestamp Timestamp of the most recent archive"
echo "# TYPE borg_last_archive_timestamp gauge"
echo "borg_last_archive_timestamp${LABELS} $LAST_ARCHIVE_TIMESTAMP"

echo "# HELP borg_last_archive_files Number of files of the most recent archive"
echo "# TYPE borg_last_archive_files gauge"
echo "borg_last_archive_files${LABELS} ${LAST_ARCHIVE_FILES}"

echo "# HELP borg_last_archive_size_bytes Original size of of the most recent archive"
echo "# TYPE borg_last_archive_size_bytes gauge"
echo "borg_last_archive_size_bytes${LABELS} ${LAST_ARCHIVE_ORIGINAL_SIZE}"

echo "# HELP borg_last_archive_deduplicated_size_bytes Deduplicated size of of the most recent archive"
echo "# TYPE borg_last_archive_deduplicated_size_bytes gauge"
echo "borg_last_archive_deduplicated_size_bytes${LABELS} ${LAST_ARCHIVE_DEDUPLICATED_SIZE}"

echo "# HELP borg_last_archive_compressed_size_bytes Compressed size of of the most recent archive"
echo "# TYPE borg_last_archive_compressed_size_bytes gauge"
echo "borg_last_archive_compressed_size_bytes${LABELS} ${LAST_ARCHIVE_COMPRESSED_SIZE}"

echo "# HELP borg_last_archive_duration_seconds Create duration of the most recent backup"
echo "# TYPE borg_last_archive_duration_seconds gauge"
echo "borg_last_archive_duration_seconds${LABELS} ${LAST_ARCHIVE_DURATION}"

[grantbevis]

Whilst it is only a few mb extra the issue is where does it end, I'm not saying yes or no here just trying to draw the line of which additional packages are added. What specific tool from coreutils are you missing @psi-4ward ?

Whilst in the past I've used images where you can add $EXTRA_PKGS or run arbitrary scripts on init this opens a new can of worms.

I would say jq is non-essential in this images use case as @toastie89 said above they've used grep to the same extent.

[psi-4ward]

Cut

[grantbevis]

I think the $EXTRA_PKGS feature will be something we'd look to implement in the extended image when ready