Log management

[felixschndr]

I was wondering how you manage the logs of Borg and especially deleting them.
I have a script which runs daily and starts Borg. I keep the logs of this script in a folder which becomes pretty crowded now and already (after about a month) has about 300 MB in size. I was thinking about how and which logs to delete. I guess after pruning my archives I could iterate over all log files and check it its corresponding backup is still present. If not delete the log file since I don't need logs of backups which don't exists anymore. I guess I could name the logs exactly the same/pretty similar as the archives itself which would make matching pretty easy.

How do you do this? Do you just keep all the logs (how much space does that need)? Do you manually clean up every once in a while? Do you have some automation set up?

[ThomasWaldmann]

The usual way to deal with that is logrotate (see /etc/logrotate.d on Linux and other POSIX OSes).

But yeah, that would be just age-based and have its own rules, not keeping the logs in sync with actually existing archives.

If you want better, write a script that uses borg list --short to get the archive names so it can know what logs to keep (and just name the logs same or similar to the archive names).

[felixschndr]

But yeah, that would be just age-based and have its own rules, not keeping the logs in sync with actually existing archives.

Yeah, that's what is important to me.

For anyone finding this in the future, this is what I've implemented:

readonly ARCHIVE_NAME=$(date +'%Y-%m-%dT%H:%M:%S')

readonly LOG_FOLDER="${SERVER_ORDNER}/Backup/logs/"
readonly LOG_FILE="${LOG_FOLDER}/${ARCHIVE_NAME}"
exec &> >(tee "${LOG_FILE}")

[...]

borg create [...] "${BORG_REPOSITORY}"::"${ARCHIVE_NAME}" 

[...]

borg prune [...] "${BORG_REPOSITORY}"

[...]

borg compact [...] "${BORG_REPOSITORY}"

[...]

mapfile -t all_backups < <(borg list --short "${BORG_REPOSITORY}")
for file in "${LOG_FOLDER}"*; do
    if [[ ! ${all_backups[*]} =~ $(basename "${file}") ]]; then
        rm --verbose "${file}"
    fi
done

This way all archives in the repo have the same name as their log. After pruning and deleting all old archives all logfiles get checked if their archive is still present. If not delete the logfile