I just released Catfood.Shapefile 1.60. This contains a fix from Libor Weigl that factors out the enumerator so that you can still access the shapefile after enumeration.
Catfood.Shapefile is a .NET library for parsing ESRI Shapefiles.
Autumn starts now north of the Equator, Spring to the south. Rendered in Catfood Earth.
(Published to the Fediverse as: Autumnal Equinox 2019 #code #catfood #earth #equinox #autumnal The start of Autumn 2019 rendered in Catfood Earth )
I've just tidied up and released a tool I've used for a while to sort photos and videos. It does a pretty good job figuring out the date each was taken and then moves them to a year + month subfolder. The source code and a binary release are now available on github - see photo-sorter.
This is a command line application with two arguments, a source folder and a destination folder. Use it like this (paths are examples and note that if there are spaces then the entire argument needs to be in quotes):
PhotoSorter.exe "C:\Users\My Name\Google Drive\Google Photos" "C:\Users\My Name\Photos"
This will process all files in the source folder, including subfolders, even if they are not photos or videos. Each file will be moved to a year + month subfolder in the destination (i.e. 2019-08) or to a special subfolder (An Unknown Date) for any files where the date the photo or video was taken cannot be determined.
In addition to moving files the tool also handles de-duplication. If the file already exists in the destination folder it is just deleted from the source and not moved. This is checked by file contents (hash) and not by name. If a different file with the same name already exists in the destination folder then PhotoSorter will move it to a unique, new filename.
I originally wrote this to handle my 'Google Photos' folder - when this feature worked it just dumped everything from Google Photos into one Drive folder with no organization. I used this periodically to tidy everything into my Photos folder also backed up to Google Drive. Now that Google has stopped syncing Drive and Photos this is still useful, especially with my script that copies new photos over to Google Drive.
Photo Sorter has been updated to handle some duplicates I've been developing in Google Photos. These are pretty specific rules but might be helpful if you are trying to maintain a local archive from Google Photos via Google Takeout. You can get the latest binary and source from github (or fork away if it's not quite what you need).
The first change is that Photo Sorter now checks for duplicates in the source folder as well as the destination. If two source files have the same date taken and the same filename then the larger file is chosen as the winner and the smaller file is deleted. The filename check ignores anything in parentheses, so 123.jpg is considered to be the same filename as 123(1).jpg. This helps alleviate a fun bug where Google Photos will export via the API a different file that was originally uploaded. I've stopped using the Google Photos API for this reason, and because it will under no circumstances allow you to download a video that is the same quality as the original upload. Crazy edge case Google. Happily Google Takeout still works so I'm stuck doing it slowly and wastefully.
The second change is that if a source duplicate is found using the rules above then it will also be deleted from the destination folder (in order to be replaced by the presumed better version of itself).
Photo Sorter copies some folder full of photos and movies to a different folder with a clean structure and some de-duplication. It's been keeping me sane since 2018.
Photo sorter has been updated to skip metadata when comparing JPEG files.
I've been picking up some duplicates when I have both a local copy and a version downloaded from Google Photos. Google Photos knocks out some metadata and so the files look different even though the photo is the same. If you've used Photo Sorter before you'll need to run it over everything again to knock out any copies.
(Published to the Fediverse as: Photo Sorter #code #photosorter #jpeg Windows command line tool to de-duplicate and sort photos by year and month, available as source and binary on GitHub. )
Warning - I no longer recommend using this script to backup Google Photos. The Google Photos API has too many bugs that Google doensn't seem interested in fixing. My personal approach at this point is to use Google Takeout to get a periodic archive of my most recent year of photos and videos. I have a tool that does some deduplication and puts everything in year/month folders. See Photo Sorter for more details.
Google has decided that backing up your photos via Google Drive is 'confusing' and so Drive based backup is going away this month. I love Google Photos but I don't trust it - I pull everything into Drive and then I stick a monthly backup from there onto an external drive in a fire safe. There is a way to get Drive backup working again using Google Apps Script and the Google Photos API. There are a few steps to follow but it's pretty straightforward - you should only need to change two lines in the script to get this working for your account.
First two four caveats to be aware of. Apps Script has a time limit and so it's possible that it could fail if moving a large number of photos. You should get an email if the script ever fails so watch out for that. Secondly and more seriously you could end up with two copies of your photos. If you use Backup and Sync to add photos from Google Drive then these photos will be downloaded from Google Photos by the script and added to Drive again. You need to either upload directly to Google Photos (i.e. from the mobile app or web site) or handle the duplicates in some way. If you run Windows then I have released a command line tool that sorts photos into year+month taken folders and handles de-duplication.
One more limitation. After a comment by 'Logan' below I realized that Apps Script has a 50MB limitation for adding files to Google Drive. The latest version of the script will detect this and send you an email listing any files that could not be copied automatically.
And a fourth limitation after investigating a comment by 'Tim' it turns out there is a bug in the Google Photos API that means it will not download original quality video files. You get some lower resolution version instead. Together with the file size limit this is a bit of a deal breaker for most videos. Photos will be fine, but videos will need a different fix.
On to the script. In Google Drive create a new spreadsheet. This is just a host for the script and makes it easy to authorize it to access Google Photos. Select 'Script editor' from the Tools menu to create a new Apps Script project.
In the script editor select 'Libraries...' from the Resources menu. Enter 1B7FSrk5Zi6L1rSxxTDgDEUsPzlukDsi4KGuTMorsTQHhGBzBkMun4iDF next to 'Add a library' and click add. This will find the Google OAuth2 library Pick the most recent version and click Save.
Select 'Project properties' from the File menu and copy the Script ID (a long sequence of letters and numbers). You'll need this when configuring the Google Photos API.
In a new window open the Google API Console, part of the Google Cloud Platform. Create a new project, click Enable APIs and Services and find and enable the Google Photos API. Then go to the Keys section and create an OAuth Client ID. You'll need to add a consent screen, the only field you need to fill out is the product name. Choose Web Application as the application type. When prompted for the authorized redirect URL enter https://script.google.com/macros/d/{SCRIPTID}/usercallback and replace {SCRIPTID} with the Script ID you copied above. Copy the Client ID and Client Secret which will be used in the next step.
Go back to the Apps Script project and paste the code below into the Code.gs window:
Enter the Client ID and Client Secret inside the empty quotes at the top of the file. You also need to add an email address to receive alerts for large files. There is a BackupFolder option at the top as well - the default is 'Google Photos' which will mimic the old behavior. You can change this if you like but make sure that the desired folder exists before running the script. Save the script.
Go back to the spreadsheet you created and reload. After a few seconds you will have a Google Photos Backup menu (to the right of the Help menu). Choose 'Authorize' from this menu. You will be prompted to give the script various permissions which you should grant. After this a sidebar should appear on the spreadsheet (if not choose 'Authorize' from the Google Photos Backup menu again). Click the authorize link from the sidebar to grant access to Google Photos. Once this is done you should be in business - choose Backup Now from the Google Photos Backup menu and any new items from yesterday should be copied to the Google Photos folder in Drive (or the folder you configured above if you changed this).
Finally you should set up a trigger to automate running the script every day. Choose 'Script editor' from the Tools menu to re-open the script, and then in the script window choose 'Current project's triggers' from the Edit menu. This will open yet another window. Click 'Add Trigger' which is cunningly hidden at the bottom right of the window. Under 'Choose which function to run' select 'runBackup'. Then under 'Select event source' select 'Time-driven'. Under 'Select type of time based trigger' select 'Day timer'. Under 'Select time of day' select the time window that works best for you. Click Save. The backup should now run every day.
The way the script is written you'll get a backup of anything added the previous day each time it runs. If there are any duplicate filenames in the backup folder the script will save a new copy of the file with (1) appended in front of the filename. Let me know in the comments if you use this script or have any suggestions to improve it.
(Published to the Fediverse as: How to backup Google Photos to Google Drive automatically after July 2019 with Apps Script #code #software #photos #appsscript #google #sheets #drive #api Easy to configure apps script to continue backing up your Google Photos to Google Drive after the July 2019 change. You just need to change two lines of the script to get this running with your Google Photos account. )
Summer starts now in the Northern Hemisphere, Winter if you happen to be south of the Equator. Rendered in Catfood Earth.
(Published to the Fediverse as: Summer Solstice 2019 #code #solstice #summer #winter #earth #northern #estival Summer Solstice 2019 as rendered by Catfood Earth. )
Catfood Earth 3.46 is now available for download. Catfood Earth for Android 1.70 is available in the Google Play Store and will update automatically if you already have it installed.
This follows hot on the heels of the last release as the new clouds layer service running on this blog can update far more frequently than the source used prior to 3.45. You will now get a fresh helping of clouds every hour! Unrelated to this release I've improved the quality of the clouds image as well. If you're interested you can read about this in exhaustive detail here.
Last weekend I rushed out a new version of Catfood Earth because the clouds layer stopped working. I'd been using xplanet clouds which published a free 2048x1024 image infrequently and for some reason the site has vanished (probably because it was based on data from the Dundee Satellite Receiving Station which has apparently closed). The University of Wisconsin-Madison Space Science and Engineering Center makes global cloud data available for free for non-commercial use so I plugged this in to get Catfood Earth up and running.
Clouds started working again last Saturday but they were slightly out of position. With a pixel on a HD screen being up to 13 miles this isn't ideal. I hope nobody is using Catfood Earth for weather forecasting or anything too serious but this bothered me enough that I had to fix it. From today the clouds layer is as close to perfect as I can make it.
Mostly for my own sanity in case I ever need to fix this again here's the process.
I use the globalir product because it covers both day and night. This is available in a number of resolutions and you just need to download a set of tiles and stitch them together - I'm using 4096x4096 as the source for the new layer. Catfood Earth uses equirectangular images which have a 2:1 aspect ratio so to start with I resized this less at the equator and more at the poles and got an image that was almost right.
The source image is Spherical (or Web) Mercator which is useful for Google Maps but not right for Catfood Earth. I found a Stack Exchange post which pointed me in the right direction - use GDAL (Geospatial Data Abstraction Library) to transform the Mercator image to equirectangular by tagging the corners and then warping it. This got me closer but it still wasn't right. More googling led me to a post by Robert Simmon with a gdalwarp tip to use -te and -te_srs to force the output to match the dimensions of the input.
The last problem is that the Mercator image doesn't reach all the way to the poles. In this case there is no alternative but to invent some data and so I flip the top and bottom 50 pixels to cover the gap. You can see this if you look carefully but as this region is rendered over ice in Catfood Earth it won't make a difference in practice (and the old image used this trick as well for a greater proportion of the clouds).
One final visual tweak is to drop the lowest intensity pixels which end up blurring out the background in Catfood Earth. You can adjust the transparency and color used to render the layer to suit your tastes (the default is 50% transparency and a gray color for the clouds which I think strikes a good balance).
I also increased the frequency of updates so a new image is available every hour. I said last week that an advantage of this new system was that I wouldn't need to push a new version of Catfood Earth to tweak the cloud logic. That's partly true - you'll get the new improved image with the current version, but I stepped down the update frequency to once a day when I started using xplanet clouds. There will be another update soon to the Windows and Android versions to switch to hourly updates.
Updated 2022-11-05 12:28:
A couple of updates here. The first is that RealEarth started watermarking images over a certain resolution and/or usage volume. You now need an API key and potentially to pay based on usage. More details here. I managed to stay in the free tier by asking nicely and scaling back to HD. The clouds download is still 4k but this is upscaled from a HD version.
Grabbing the latest image (for globalir) worked until sometime in October and then the tiles stopped lining up correctly. I'm not sure what changed, I guess that 'latest' used to mean the latest composite and now means the latest data for any given tile regardless of it is the same time as the others. I just fixed this to use the latest complete image (this API provides available dates and times, and then you need to use the date and time in the globalir call to get a working composite image).
Updated 2023-09-03 01:03:
There is now a video made from the last 48 hours of cloud images here, updated hourly. The video skips the step of dropping low intensity pixels so you get 100% of the cloud cover.
(Published to the Fediverse as: Improving the accuracy of the new Catfood Earth clouds layer #code #earth #clouds Using the Geospatial Data Abstraction Library to transform a Spherical Mercator cloud image to equirectangular for Catfood Earth. )
Catfood Earth 3.45 is now available to download. Catfood Earth for Android 1.60 is available on Google Play and will update automatically if you have it installed.
I only just released 3.44 with some timezone updates but in the past week the location I had been using for global cloud cover abruptly shut down. If you like up to date clouds you'll want to install the new versions as soon as possible. With this update I'm building a cloud image every three hours and serving through this blog (and thankfully CloudFlare) so any further changes should not require a code release.
Spring starts now in the Northern Hemisphere. Rendered in (the recently updated) Catfood Earth.
(Published to the Fediverse as: Vernal (Spring) Equinox 2019 #code #earth #equinox #spring #autumn #vernal Spring or Vernal Equinox 2019 at 21:58 UTC on Match 20, 2019. Rendered in Catfood Earth. )
Catfood Earth 3.44 is now available to download.
The timezone database has been updated to 2018i.
Eric Muller's shapefile map of timezones is no longer maintained and so Catfood Earth has switched to Evan Siroky's timezone boundary builder version.
A bug that could cause all volcanoes to be plotted at 0,0 depending on your system locale has been fixed.
Download a Sharepoint File with GraphServiceClient (Microsoft Graph API)
Export Google Fit Daily Steps, Weight and Distance to a Google Sheet
Monitor page index status with Google Sheets, Apps Script and the Google Search Console API
Scanning from the ADF using WIA in C#
Enable GZIP compression for Amazon S3 hosted website in CloudFront
Sending email via GMail in C#/.NET using SmtpClient
Which PG&E rate plan works best for EV charging?
Reading and Writing Office 365 Excel from a Console app using the Microsoft.Graph C# Client API