Improving the accuracy of the new Catfood Earth clouds layer
Last weekend I rushed out a new version of Catfood Earth because the clouds layer stopped working. I'd been using xplanet clouds which published a free 2048x1024 image infrequently and for some reason the site has vanished (probably because it was based on data from the Dundee Satellite Receiving Station which has apparently closed). The University of Wisconsin-Madison Space Science and Engineering Center makes global cloud data available for free for non-commercial use so I plugged this in to get Catfood Earth up and running.
Clouds started working again last Saturday but they were slightly out of position. With a pixel on a HD screen being up to 13 miles this isn't ideal. I hope nobody is using Catfood Earth for weather forecasting or anything too serious but this bothered me enough that I had to fix it. From today the clouds layer is as close to perfect as I can make it.
Mostly for my own sanity in case I ever need to fix this again here's the process.
I use the globalir product because it covers both day and night. This is available in a number of resolutions and you just need to download a set of tiles and stitch them together - I'm using 4096x4096 as the source for the new layer. Catfood Earth uses equirectangular images which have a 2:1 aspect ratio so to start with I resized this less at the equator and more at the poles and got an image that was almost right.
The source image is Spherical (or Web) Mercator which is useful for Google Maps but not right for Catfood Earth. I found a Stack Exchange post which pointed me in the right direction - use GDAL (Geospatial Data Abstraction Library) to transform the Mercator image to equirectangular by tagging the corners and then warping it. This got me closer but it still wasn't right. More googling led me to a post by Robert Simmon with a gdalwarp tip to use -te and -te_srs to force the output to match the dimensions of the input.
The last problem is that the Mercator image doesn't reach all the way to the poles. In this case there is no alternative but to invent some data and so I flip the top and bottom 50 pixels to cover the gap. You can see this if you look carefully but as this region is rendered over ice in Catfood Earth it won't make a difference in practice (and the old image used this trick as well for a greater proportion of the clouds).
One final visual tweak is to drop the lowest intensity pixels which end up blurring out the background in Catfood Earth. You can adjust the transparency and color used to render the layer to suit your tastes (the default is 50% transparency and a gray color for the clouds which I think strikes a good balance).
I also increased the frequency of updates so a new image is available every hour. I said last week that an advantage of this new system was that I wouldn't need to push a new version of Catfood Earth to tweak the cloud logic. That's partly true - you'll get the new improved image with the current version, but I stepped down the update frequency to once a day when I started using xplanet clouds. There will be another update soon to the Windows and Android versions to switch to hourly updates.
Updated 2022-11-05 12:28:
A couple of updates here. The first is that RealEarth started watermarking images over a certain resolution and/or usage volume. You now need an API key and potentially to pay based on usage. More details here. I managed to stay in the free tier by asking nicely and scaling back to HD. The clouds download is still 4k but this is upscaled from a HD version.
Grabbing the latest image (for globalir) worked until sometime in October and then the tiles stopped lining up correctly. I'm not sure what changed, I guess that 'latest' used to mean the latest composite and now means the latest data for any given tile regardless of it is the same time as the others. I just fixed this to use the latest complete image (this API provides available dates and times, and then you need to use the date and time in the globalir call to get a working composite image).
