Autumnal Equinox 2019

Updated on Sunday, May 3, 2020

Autumnal Equinox 2019 in Catfood Earth

Autumn starts now north of the Equator, Spring to the south. Rendered in Catfood Earth.

(Previously)

Photo Sorter 1.00

Updated on Sunday, January 31, 2021

Bacterial Mat

I've just tidied up and released a tool I've used for a while to sort photos and videos. It does a pretty good job figuring out the date each was taken and then moves them to a year + month subfolder. The source code and a binary release are now available on github - see photo-sorter.

This is a command line application with two arguments, a source folder and a destination folder. Use it like this (paths are examples and note that if there are spaces then the entire argument needs to be in quotes):

PhotoSorter.exe "C:\Users\My Name\Google Drive\Google Photos" "C:\Users\My Name\Photos"

This will process all files in the source folder, including subfolders, even if they are not photos or videos. Each file will be moved to a year + month subfolder in the destination (i.e. 2019-08) or to a special subfolder (An Unknown Date) for any files where the date the photo or video was taken cannot be determined.

In addition to moving files the tool also handles de-duplication. If the file already exists in the destination folder it is just deleted from the source and not moved. This is checked by file contents (hash) and not by name. If a different file with the same name already exists in the destination folder then PhotoSorter will move it to a unique, new filename.

I originally wrote this to handle my 'Google Photos' folder - when this feature worked it just dumped everything from Google Photos into one Drive folder with no organization. I used this periodically to tidy everything into my Photos folder also backed up to Google Drive. Now that Google has stopped syncing Drive and Photos this is still useful, especially with my script that copies new photos over to Google Drive.

Updated 2021-01-31 11:59:

See Photo Sorter 1.10, updated to skip metadata when comparing JPEG files.

How to backup Google Photos to Google Drive automatically after July 2019 with Apps Script

Updated on Sunday, March 14, 2021

Google Photos backup to Google Drive shutting down in July 2019

Google has decided that backing up your photos via Google Drive is 'confusing' and so Drive based backup is going away this month. I love Google Photos but I don't trust it - I pull everything into Drive and then I stick a monthly backup from there onto an external drive in a fire safe. There is a way to get Drive backup working again using Google Apps Script and the Google Photos API. There are a few steps to follow but it's pretty straightforward - you should only need to change two lines in the script to get this working for your account.

First two four caveats to be aware of. Apps Script has a time limit and so it's possible that it could fail if moving a large number of photos. You should get an email if the script ever fails so watch out for that. Secondly and more seriously you could end up with two copies of your photos. If you use Backup and Sync to add photos from Google Drive then these photos will be downloaded from Google Photos by the script and added to Drive again. You need to either upload directly to Google Photos (i.e. from the mobile app or web site) or handle the duplicates in some way. If you run Windows then I have released a command line tool that sorts photos into year+month taken folders and handles de-duplication.

One more limitation. After a comment by 'Logan' below I realized that Apps Script has a 50MB limitation for adding files to Google Drive. The latest version of the script will detect this and send you an email listing any files that could not be copied automatically.

And a fourth limitation after investigating a comment by 'Tim' it turns out there is a bug in the Google Photos API that means it will not download original quality video files. You get some lower resolution version instead. Together with the file size limit this is a bit of a deal breaker for most videos. Photos will be fine, but videos will need a different fix.

On to the script. In Google Drive create a new spreadsheet. This is just a host for the script and makes it easy to authorize it to access Google Photos. Select 'Script editor' from the Tools menu to create a new Apps Script project.

In the script editor select 'Libraries...' from the Resources menu. Enter 1B7FSrk5Zi6L1rSxxTDgDEUsPzlukDsi4KGuTMorsTQHhGBzBkMun4iDF next to 'Add a library' and click add. This will find the Google OAuth2 library Pick the most recent version and click Save.

Select 'Project properties' from the File menu and copy the Script ID (a long sequence of letters and numbers). You'll need this when configuring the Google Photos API.

In a new window open the Google API Console, part of the Google Cloud Platform. Create a new project, click Enable APIs and Services and find and enable the Google Photos API. Then go to the Keys section and create an OAuth Client ID. You'll need to add a consent screen, the only field you need to fill out is the product name. Choose Web Application as the application type. When prompted for the authorized redirect URL enter https://script.google.com/macros/d/{SCRIPTID}/usercallback and replace {SCRIPTID} with the Script ID you copied above. Copy the Client ID and Client Secret which will be used in the next step.

Go back to the Apps Script project and paste the code below into the Code.gs window:

Enter the Client ID and Client Secret inside the empty quotes at the top of the file. You also need to add an email address to receive alerts for large files. There is a BackupFolder option at the top as well - the default is 'Google Photos' which will mimic the old behavior. You can change this if you like but make sure that the desired folder exists before running the script. Save the script.

Go back to the spreadsheet you created and reload. After a few seconds you will have a Google Photos Backup menu (to the right of the Help menu). Choose 'Authorize' from this menu. You will be prompted to give the script various permissions which you should grant. After this a sidebar should appear on the spreadsheet (if not choose 'Authorize' from the Google Photos Backup menu again). Click the authorize link from the sidebar to grant access to Google Photos. Once this is done you should be in business - choose Backup Now from the Google Photos Backup menu and any new items from yesterday should be copied to the Google Photos folder in Drive (or the folder you configured above if you changed this).

Finally you should set up a trigger to automate running the script every day. Choose 'Script editor' from the Tools menu to re-open the script, and then in the script window choose 'Current project's triggers' from the Edit menu. This will open yet another window. Click 'Add Trigger' which is cunningly hidden at the bottom right of the window. Under 'Choose which function to run' select 'runBackup'. Then under 'Select event source' select 'Time-driven'. Under 'Select type of time based trigger' select 'Day timer'. Under 'Select time of day' select the time window that works best for you. Click Save. The backup should now run every day.

The way the script is written you'll get a backup of anything added the previous day each time it runs. If there are any duplicate filenames in the backup folder the script will save a new copy of the file with (1) appended in front of the filename. Let me know in the comments if you use this script or have any suggestions to improve it.

More Google Apps Script Projects

Summer Solstice 2019

Summer Solstice 2019 in Catfood Earth

Summer starts now in the Northern Hemisphere, Winter if you happen to be south of the Equator. Rendered in Catfood Earth.

(Previously)

Catfood Earth 3.46

Improved clouds in Catfood Earth

Catfood Earth 3.46 is now available for download. Catfood Earth for Android 1.70 is available in the Google Play Store and will update automatically if you already have it installed.

This follows hot on the heels of the last release as the new clouds layer service running on this blog can update far more frequently than the source used prior to 3.45. You will now get a fresh helping of clouds every hour! Unrelated to this release I've improved the quality of the clouds image as well. If you're interested you can read about this in exhaustive detail here.

Improving the accuracy of the new Catfood Earth clouds layer

Equirectangular clouds for Catfood Earth

Last weekend I rushed out a new version of Catfood Earth because the clouds layer stopped working. I'd been using xplanet clouds which published a free 2048x1024 image infrequently and for some reason the site has vanished (probably because it was based on data from the Dundee Satellite Receiving Station which has apparently closed). The University of Wisconsin-Madison Space Science and Engineering Center makes global cloud data available for free for non-commercial use so I plugged this in to get Catfood Earth up and running.

Clouds started working again last Saturday but they were slightly out of position. With a pixel on a HD screen being up to 13 miles this isn't ideal. I hope nobody is using Catfood Earth for weather forecasting or anything too serious but this bothered me enough that I had to fix it. From today the clouds layer is as close to perfect as I can make it.

Mostly for my own sanity in case I ever need to fix this again here's the process.

I use the globalir product because it covers both day and night. This is available in a number of resolutions and you just need to download a set of tiles and stitch them together - I'm using 4096x4096 as the source for the new layer. Catfood Earth uses equirectangular images which have a 2:1 aspect ratio so to start with I resized this less at the equator and more at the poles and got an image that was almost right.

The source image is Spherical (or Web) Mercator which is useful for Google Maps but not right for Catfood Earth. I found a Stack Exchange post which pointed me in the right direction - use GDAL (Geospatial Data Abstraction Library) to transform the Mercator image to equirectangular by tagging the corners and then warping it. This got me closer but it still wasn't right. More googling led me to a post by Robert Simmon with a gdalwarp tip to use -te and -te_srs to force the output to match the dimensions of the input.

The last problem is that the Mercator image doesn't reach all the way to the poles. In this case there is no alternative but to invent some data and so I flip the top and bottom 50 pixels to cover the gap. You can see this if you look carefully but as this region is rendered over ice in Catfood Earth it won't make a difference in practice (and the old image used this trick as well for a greater proportion of the clouds).

One final visual tweak is to drop the lowest intensity pixels which end up blurring out the background in Catfood Earth. You can adjust the transparency and color used to render the layer to suit your tastes (the default is 50% transparency and a gray color for the clouds which I think strikes a good balance).

I also increased the frequency of updates so a new image is available every hour. I said last week that an advantage of this new system was that I wouldn't need to push a new version of Catfood Earth to tweak the cloud logic. That's partly true - you'll get the new improved image with the current version, but I stepped down the update frequency to once a day when I started using xplanet clouds. There will be another update soon to the Windows and Android versions to switch to hourly updates.

Catfood Earth 3.45

New cloud layer image in Catfood Earth

Catfood Earth 3.45 is now available to download. Catfood Earth for Android 1.60 is available on Google Play and will update automatically if you have it installed.

I only just released 3.44 with some timezone updates but in the past week the location I had been using for global cloud cover abruptly shut down. If you like up to date clouds you'll want to install the new versions as soon as possible. With this update I'm building a cloud image every three hours and serving through this blog (and thankfully CloudFlare) so any further changes should not require a code release.

Vernal (Spring) Equinox 2019

Spring Equinox 2019 in Catfood Earth

Spring starts now in the Northern Hemisphere. Rendered in (the recently updatedCatfood Earth.

(Previously)

Catfood Earth 3.44

Updated timezones in Catfood Earth 3.44

Catfood Earth 3.44 is now available to download.

The timezone database has been updated to 2018i.

Eric Muller's shapefile map of timezones is no longer maintained and so Catfood Earth has switched to Evan Siroky's timezone boundary builder version.

A bug that could cause all volcanoes to be plotted at 0,0 depending on your system locale has been fixed.

Download Catfood Earth.

(Previously)

Winter Solstice 2018

Winter Solstice 2018 in Catfood Earth

It's the start of Winter in the Northern Hemisphere (Summer on the other side of the equator). Rendered in Catfood Earth.

(PreviouslyPreviouslyPreviouslyPreviouslyPreviously)