Data Dumps not auto-generating

[cdrini]

Despite #5263 being resolved, it looks like the data dumps weren't uploaded on July 1st :/

Relevant URL?

• https://github.com/internetarchive/openlibrary/wiki/Generating-Data-Dumps
• https://archive.org/details/ol_exports?sort=-publicdate

Related issues and pull requests:

• Data dump for reading log stats #3989
• 2021-01 datadumps smaller than expected (missing data?) #4621
• 2021-01-31 combined data dump missing data/out of date data #4671
• ol-home0 provisioning needs rsync ferm/input #4723
• Add helper eg /data/ol_dump_editions_latest.txt.gz links for ratings/readinglog #5546
• Manually kick data dump for 2021-09 #5673
• Manually kick data dump for 2021-10 #5719
• Fix cron job for oldump and sitemaps #5892 - Worth reading!
• Datadumps: logger is blocking the cron job #6158
• Fix shebang lines in scripts/oldump.py and scripts/sitemaps/sitemaps.py  #6163

Related files:

• docker-compose.production.yml defines cron-jobs Docker container.
• docker/ol-cron-start.sh sets up the cron tasks.
• olsystem: /etc/cron.d/openlibrary.ol_home0 defines the actual job
  • modify and then to reactivate do: crontab /etc/cron.d/openlibrary.ol_home0 Also: https://cron.help
  • internetarchive/olsystem#140
• scripts/oldump.sh is the script that gets run.
  • Data dumps: chmod +x scripts/oldump.sh #5860

Proposal & Constraints

• Run manually for now

Related files

Stakeholders

@mekarpeles @jimman2003

[cdrini]

It looks like they were generated, but the file names look off; there's no date. And the --archive flag which is used to... you know upload the data dumps, somehow made it into the filename. So I imagine the upload never happened.

On ol-home0:

cd /opt/openlibrary

COMPOSE_FILE="docker-compose.yml:docker-compose.production.yml" \
HOSTNAME="$HOSTNAME" \
docker-compose run \
    -u openlibrary \
    -e "SCRIPTS=/openlibrary/scripts" \
    cron-jobs \
    bash

cd /1/var/tmp/dumps
openlibrary@ol-home0:/1/var/tmp/dumps$ du -sh *
4.0K    ol_cdump_--archive.txt.gz
4.0K    ol_dump_--archive.txt.gz

^ Although the file sizes definitely don't look correct here.

[cdrini]

Going to follow the steps here to generate a new dump: #4621 (comment)

Had to remove the old files manually; complaining about permissions... sudo rm -rf /1/var/tmp/dumps/ol_*

Then (from the above comment):

tmux
# ---
cd /opt/openlibrary

COMPOSE_FILE="docker-compose.yml:docker-compose.production.yml" \
HOSTNAME="$HOSTNAME" \
docker-compose run \
    -u openlibrary \
    -e "SCRIPTS=/openlibrary/scripts" \
    cron-jobs \
    bash
# ---
cd /1/var/tmp
/olsystem/bin/cron/oldump.sh ol-db1 openlibrary `date -d yesterday '+%Y-%m-%d'` --archive

Waiting...

[cdrini]

New dump created 👍

[cdrini]

Kicked off manual dump; see #3989 (comment)