2021-01 datadumps smaller than expected (missing data?)

[mekarpeles]

Related to #4671, #4723

Looks like archives for January 31 2021 are much smaller than usually comparing to months past. Is it possible to re-upload them with missing data?

https://archive.org/download/ol_dump_2021-01-31

Past months which is what the sizes are typically:

https://archive.org/download/ol_dump_2020-12-31
https://archive.org/download/ol_dump_2020-11-30

• Compare 2020 dumps vs. 2021 dumps
  • Similar filenames
  • Similar file sizes
  • Similar file contents
• Ensure that rsync runs properly on the end-of-March 2021 transfer

Evidence / Screenshot (if possible)

Relevant url?

Steps to Reproduce

1. Go to ...
2. Do ...

• Actual:
• Expected:

Details

• Logged in (Y/N)?
• Browser type/version?
• Operating system?
• Environment (prod/dev/local)? prod

Proposal & Constraints

Related files

Stakeholders

[hornc]

@mekarpeles @cclauss

I just got burned by this issue when downloading the Feb dumps from https://archive.org/download/ol_dump_2021-02-28

I was looking to update the orphaned edition counts, that data shows 4M, which is double the last proper count. That gave me a scare :)

The data looks like it is simply out of date, the most recent last modified date in the editions dump is 2019-03, which matches the orphan count from then. I'm not sure where this 2019 data is being pulled from, but hopefully that'll make it easier for you to track down.

[cclauss]

This issue was fixed in https://archive.org/download/ol_dump_2021-03-19 but let's leave it until the April 1st dump.

[hornc]

@cclauss , did you see my comments in slack that this latest dump has some old versions of some items?
Checking for orphaned editions shows that there are 300K more orphans than the last dump, but I think this is due to bad data in the dump.

An example is this edition, currently at a v=2 since 2009, but is in the dump with latest version v=1
e.g. https://openlibrary.org/books/OL10003370M/Discussion_Paper_On_Registration_Of_Rights_In_Security_By_Companies

Here's another one that is one version behind in the data dump: /books/OL10024097M v=3 in the dump vs. v=4 on the site

Many / (most?) of the entries in the data dump have the current version data. e.g. /books/OL10002616M , v=4 in the data, v=4 on the live site