How to skip photos already uploaded?

[simonhf]

Unfortunately the upload failed for me after about 1,100 photos:

$ python flickr-restore.py config.json
INFO     Uploading photo: '18812229225: /home/simon/20210406-flickr/unzipped/18812229225_6517b32507_o.jpg'
Traceback (most recent call last):
  File "flickr-restore.py", line 193, in <module>
    main(config)
  File "flickr-restore.py", line 170, in main
    uploader.upload_all_albums()
  File "flickr-restore.py", line 55, in upload_all_albums
    self.upload_album(album)
  File "flickr-restore.py", line 80, in upload_album
    self.upload_photo(flickr_photo_id, album["id"], flickr_photo_id == cover_photo_id)
  File "flickr-restore.py", line 154, in upload_photo
    self.session.post("https://photoslibrary.googleapis.com/v1/mediaItems:batchCreate", json=create_request_body).raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://photoslibrary.googleapis.com/v1/mediaItems:batchCreate

The Google photo API also seemed to be within limit:

Another help page says:

In addition to these limits there are other quotas that exist to protect the reliability and integrity of our systems. If you receive a 429 error, you've likely hit one of these quotas. If you receive one of these errors, reduce the number of requests you are making over a given time period. For more information, refer to retrying failed requests and exponential backoff.

I tried restarting the script but it then starts uploading the same photos AGAIN which is uploaded before, thus eating further into my limit :-(

How to ask the script to not re-upload previously uploaded photos?

[simonhf]

Seems like it only re-uploaded the first album for some reason. On the other albums then it says, e.g.:

$ python flickr-restore.py config.json
...
INFO     Going to upload: 'Camera Uploads' [3/39]
INFO     Found existing album: 'Camera Uploads'
INFO     Skipping upload of album 'Camera Uploads' since item count in Google [1007] is >= item count in Flickr [1007]
INFO     Going to upload: 'Originals' [4/39]
INFO     Found existing album: 'Originals'
INFO     Starting upload of photos to: 'Originals'
...

So it seems like the script will skip complete albums but if a 1,000 photo album is missing a single photo then it will re-upload the album...

[simonhf]

Also, after restarting the script then only after ~ 100 photos or so then the same 429 error popped up again:

$ python flickr-restore.py config.json
...
Traceback (most recent call last):
  File "flickr-restore.py", line 193, in <module>
    main(config)
  File "flickr-restore.py", line 170, in main
    uploader.upload_all_albums()
  File "flickr-restore.py", line 55, in upload_all_albums
    self.upload_album(album)
  File "flickr-restore.py", line 80, in upload_album
    self.upload_photo(flickr_photo_id, album["id"], flickr_photo_id == cover_photo_id)
  File "flickr-restore.py", line 154, in upload_photo
    self.session.post("https://photoslibrary.googleapis.com/v1/mediaItems:batchCreate", json=create_request_body).raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://photoslibrary.googleapis.com/v1/mediaItems:batchCreate

Even though I'm still within quota:

[simonhf]

Update: In the end I abandoned this tool due to the HTTP 429 errors. The only alternative I found is to use Google's Backup and Sync program which uploaded all ~ 20,000 photos over night via a 650 Mbps internet connection. Only downside is that it doesn't understand albums but all the pictures are searchable using keywords, e.g. 'watch' or 'toilet' etc.

[simonhf]

Update: The Google Photos search mechanism works really well... but only for me. If I share the photos -- with e.g. my wife -- then she just sees a scrollable list of ~ 20,000 photos and no way to search them. This is frustrating and makes me wonder if Google Photos is the right way to go...

[angrycandy]

@simonhf

https://github.com/angrycandy/flickr-to-google-photos fixes a lot of the problems you encountered.

But it is slower than Backup and Sync because of the 10000 requests per day quota.