Add Open Library record for every work/edition in Internet Archive 2020 Wishlist

[mekarpeles]

There are 2.6M items on the Internet Archive Openlibraries Wishlist. We want to make sure each of these books has a corresponding catalog entry in openlibrary.org.

Steps

• Download the wishlistdataset from Add Open Library record for every work/edition in Internet Archive 2020 Wishlist #869 (@mekarpeles will provide an updated link)
• We will take the first 1,000 items in the wishlist (as a test) and see (2.a.) how many are in Open Library...
  • As a first run, we can check these 1,000 wishlist items against the Open Library books API: https://openlibrary.org/dev/docs/api/books. This won't scale to 2.6M editions which is how many are on the wishlist. So, instead, we'll need to download + process the editions data dump https://openlibrary.org/data/ol_dump_editions_latest.txt.gz

See: https://archive.org/details/open_libraries_wish_list
https://archive.org/download/open_libraries_wish_list/wish_list_isbn13_ver_1.csv.zip

@thisismattmiller -- can get a master csv which collates isbn10, isbn13, and oclc (into one row)?

We want to add all of these records into Open Library so we can query by any of these identifier fields and retrieve the metadata for the book.

Also, do we have metadata on these books?

[mekarpeles]

We may want to add all of these to a subject a la https://openlibrary.org/subjects/internet_archive_wishlist

[thisismattmiller]

All data for the wish list is found in the sqlite database in my home directory (/home/mattmiller/sqlite_database) there are example scripts in there for serialization.
Field information is found in this google sheet: https://docs.google.com/spreadsheets/d/1GDATWbgncmQzDaTJVdJU1kVcRhJIuMs0zHUnoCITED0/edit?usp=sharing

The basic SQL query to get everything that will appear on the wish list is:
SELECT * FROM data where flagged_author == 0 and flagged_publisher == 0 and no_author == 0 and no_publisher == 0

One thing to know is that the wish list tried to be as inclusive as possible, so it used the field classify_related_print_editions to add additional ISBNs (often of non-english languages) to the final list. These isbns would not have their own row in the DB but will appear in the wish list.

Basic JSON serialization of the wish list can also be found in /home/mattmiller/wish_list

[tfmorris]

We may want to add all of these to a subject a la https://openlibrary.org/subjects/internet_archive_wishlist

Noooo. No more fake subjects! They're worse than fake news. "In my little sparkly library" is not a subject. A subject is something that a book is about like "French politics" or "Himalayan mountains."

[tfmorris]

What is the basis for these 2.6 million wishes? ie where did they come from?

[mekarpeles]

@tfmorris -- got it, no subject :)

We've been using subject instead of list because our lists are unscalable/broken/challenging to fix. A new db entry is added every time a seed is added to a list causing huge db bloat. We've been experimenting w/ a new lists db (which is being used to power our "Want to Read", "Already Read", etc. Reading Log feature) which is not backed by infogami.

The list comes from here: https://blog.archive.org/2018/03/14/lets-build-a-great-digital-library-together-starting-with-a-wishlist/

[mekarpeles]

One thing to know is that the wish list tried to be as inclusive as possible, so it used the field classify_related_print_editions to add additional ISBNs (often of non-english languages) to the final list. These isbns would not have their own row in the DB but will appear in the wish list.

@thisismattmiller, is this also the case (i.e. isbn synonyms are included) in the wish_list_march_2018.ndjson?

If so, would you be able to generate a copy of the json that does not have synonyms? In order to import into Open Library, we'll need the exact book metadata / isbn, etc

[thisismattmiller]

Sorry I missed this notification.

Yes, that would be possible to exclude the synonyms. In /home/mattmiller/sqlite_database/scripts/serialize_basic.py starting on line 200 there is:

			# see if we haver other print versions available to add to this record
			if row['classify_related_print_editions'] is not None:
				row['classify_related_print_editions'] = json.loads(row['classify_related_print_editions'])
				for e in row['classify_related_print_editions']:
					for isbn in e['isbn']:
						if isbn not in added_isbn:
							if int(isbn) in have_lookup:
								skiped_isbns+=1
							else:
								# overwrite the obj and add this new one in
								obj['isbn13'] = isbn
								obj['isbn10'] = to_isbn10(isbn)
								obj['oclc'] = e['oclc']
								obj['language'] = e['language']
								added_isbn[isbn] = True
								# write it out
								out_json.write(json.dumps(obj)+'\n')
								out_csv_writer.writerow([obj['isbn13'],obj['isbn10'],obj['oclc'],obj['language'],obj['title'],obj['date']," | ".join(obj['author'])])
						else:
							# print('already added',isbn)
							pass


		else:

			#this was not a print version but there might be related print versions we have collected
			# see if we haver other print versions available to add to this record
			if row['classify_related_print_editions'] is not None:
				row['classify_related_print_editions'] = json.loads(row['classify_related_print_editions'])
				for e in row['classify_related_print_editions']:
					for isbn in e['isbn']:
						if isbn not in added_isbn:

							if int(isbn) in have_lookup:
								skiped_isbns+=1
							else:
								# overwrite the obj and add this new one in
								obj['isbn13'] = isbn
								obj['isbn10'] = to_isbn10(isbn)
								obj['oclc'] = e['oclc']
								obj['language'] = e['language']
								added_isbn[isbn] = True
								# write it out
								out_json.write(json.dumps(obj)+'\n')
								out_csv_writer.writerow([obj['isbn13'],obj['isbn10'],obj['oclc'],obj['language'],obj['title'],obj['date']," | ".join(obj['author'])])

						else:
							# print('already added',isbn)
							pass

Just comment that out and run it and it will not populate out the classify related titles aka isbn synonyms.
-Matt

[sbshah97]

@thisismattmiller what would be an easy way to separate out Editions from Works on the Wishlist such that we include just a single Work (from Multiple Editions).

[thisismattmiller]

In the SQLite DB you would need to limit it to only things that:
has_classify = 1
And then collapse rows together that have the same classify_work_id value.

The classify_work_id is a work identifier, so if two editions have the same id it is the same work.