Photo Sorter has been updated to handle some duplicates I've been developing in Google Photos. These are pretty specific rules but might be helpful if you are trying to maintain a local archive from Google Photos via Google Takeout. You can get the latest binary and source from github (or fork away if it's not quite what you need).
The first change is that Photo Sorter now checks for duplicates in the source folder as well as the destination. If two source files have the same date taken and the same filename then the larger file is chosen as the winner and the smaller file is deleted. The filename check ignores anything in parentheses, so 123.jpg is considered to be the same filename as 123(1).jpg. This helps alleviate a fun bug where Google Photos will export via the API a different file that was originally uploaded. I've stopped using the Google Photos API for this reason, and because it will under no circumstances allow you to download a video that is the same quality as the original upload. Crazy edge case Google. Happily Google Takeout still works so I'm stuck doing it slowly and wastefully.
The second change is that if a source duplicate is found using the rules above then it will also be deleted from the destination folder (in order to be replaced by the presumed better version of itself).
Photo Sorter copies some folder full of photos and movies to a different folder with a clean structure and some de-duplication. It's been keeping me sane since 2018.
Summer starts right now (09:14 UTC June 21 2022) in the northern hemisphere, winter for those with latitude signum differences. Rendered in Catfood Earth.
Just in case it helps someone else I was able to fetch an Outlook iCal feed using C#'s WebClient for years until it stopped working in June, 2022 with a 400 / bad request error. I was downloading a set of calendars and the fix was just to use a new WebClient for each calendar so it must be some kind of state thing in WebClient.
To regain around 1% of sanity I have a task that pulls the various calendars that life throws at me and combines them into a single, de-duplicated calendar. The Google Calendar on my phone is gorgeous as a result. My primary calendar is orange and then all of the miscellanea are teal and unique. It would be great if Google Calendar could do this without help, but it was worth the effort not to have some random soccer match repeated five times in different colors.
Last week one calendar, an Outlook feed, started failing with 400 bad request.
Naturally I assumed that the server had started to suddenly care about some header or other and I started playing around with setting User-Agent and various Accept headers without any luck. To make debugging slightly easier I moved the Outlook calendar out of a loop (where I was iterating through a list of iCal feeds that I need to be aware of) and then it magically started working. The magic in this case must be a fresh WebClient and so the fix was to use a new WebClient for each calendar instead of reusing a single instance. It looks like WebClient is deprecated in .NET 6 and one is supposed to start using HttpClient instead so that's probably another fix but not one I'm going to wrestle with today.
(Published to the Fediverse as:
Outlook/Office iCal feed 400 bad request error with C# WebClient #code#c##outlook How to fix a 400 / bad request error with the C# WebClient when downloading an Office iCal feed.)
After a few experiments I have a pretty decent platform for capturing long term time lapse footage. The system is designed to run for around a month and captures a photo every thirty seconds during daylight hours. It's a fairly cheap build so I'm not too worried about theft or accidental damage. This post contains build instructions and sample code to get the camera up and running. Before that, here's a quick video made from a few test runs:
On the hardware side you need an Arduino Mega 2560 REV3, the OV5642 camera module, an SD module and card, a battery and a solar panel. You'll also want a waterproof case, cables and some zip ties. Everything I used is on this Amazon list. I tried a few different power solutions and the Voltaic Systems 6W solar panel + 6,400 mAh battery combination is ideal. Their battery banks are 'always on' and continue to provide power even when the Arduino isn't doing anything. Most battery banks shut down when there is low power usage. In my testing the 6W panel managed to keep the system running with no issues.
The main limitation is that Arduino only supports up to 32GB SD cards. With the code below this allows around a month of photos. Depending on the application you could go much longer - the variables are interval (I'm shooting a frame around every 30 seconds) and file size (lower resolution and/or JPEG quality). At some point I'd like to figure out cellular upload but for now 32GB will have to do.
Assembly is pretty easy. The OV5642 camera uses both the I2C and SPI buses. For I2C connect the pins labeled SCL and SDA to SCL and SDA on the Arduino Mega (D21 and D20 pins - search Arduino Mega pinout if necessary). The SPI connections are MISO, MOSI and SCK (available at D50, D51 and D52). You also need CS (chip select) - wire this to D10. Finally wire ground and power to the GND and 5V pins. The SD module also uses SPI. Wire its MISO, MOSI and SCK to the same pins (D50, D51 and D52). SD needs a separate CS (chip select), use D7. And finally for SD connect ground and power to GND and 5V. That's it.
The box linked above is large enough to fit all the components easily. You'll need to drill two holes, one for the camera lens and one for the solar panel cable. Once assembled, seal around both as well as you can.
To program the Mega you need to install the Arduino IDE and then download and install the ArduCAM library. Make sure that the ArduCAM folder is directly under the libraries folder for your Arduino IDE (on Windows this is probably C:\Program Files (x86)\Arduino\libraries\ArduCAM). Then open the file memorysaver.h in the ArduCAM folder and make sure that only the OV5642 line is uncommented (#define OV5642_MINI_5MP_PLUS). If you get build errors it's almost certainly this step that is the problem so check carefully.
At this point I'd suggest trying the blink sample to make sure that you can connect to and program your board successfully. Once this is working create a new sketch and then copy the code below:
A few notes on the code. Near the top you can uncomment the line #define USE_SERIAL to get diagnostic messages written to the serial monitor. This can be useful for debugging if necessary. The sketch uses the built in LED to indicate problems as well - if everything is working you'll see the LED light up during setup and then switch off once the main loop starts. When a picture is written to the SD card the SD module LED will light up, so if you see that happening every 30 seconds or so then you should be in good shape. If the setup code fails to find the camera or SD module the internal LED will blink indicating that you need to fix something. Check that the OV5642 and SD modules are correctly wired. I would also try disconnecting power from the Mega for a few seconds. I have experienced connection issues with the OV5642 that are only fixed by a power cycle.
At startup there is also a check for existing files on the SD card and the file number is incremented by 1000 as needed until existing files will not be overwritten. This is useful in case the system loses power and restarts for any reason. I have included most configurable parameters for the camera module in the setup code like contrast and exposure adjustments so these should be easy to tweak based on your specific application.
Finally the startup code disables Mega features that are not used and after each photo the camera is powered down and the Mega put to sleep. Each sleep is around nine seconds. The number of sleeps depends on the size of the image - if below 200K it will sleep for around half an hour before trying again. This saves filling up the SD card with nighttime images and preserves power when the solar panel is unlikely to be helpful. This is another area to tweak depending on how you plan to use the system.
If you have any questions (or if you build this and it works for you) please leave a comment below.
(Published to the Fediverse as:
Long term solar powered time lapse camera using Arduino #code#video#arduino#camera#c++ How to build and program a solar powered time lapse camera based on the Arduino Mega 2560 and a 5MP camera module. Includes full parts list and working code.)
Catfood WebCamSaver 3.27 is available to download.
This release contains the latest webcam list and will upgrade any current set of webcams. I'm currently releasing updates for WebCamSaver every three months with the latest cams.
By Robert Ellison. Updated on Tuesday, December 6, 2022.
I love almost everything about Todoist. It's rich enough to scratch all my productivity itches while also being basic enough that I don't spend time gardening my tasks. Also, their Android app is just gorgeous, the exact opposite of Google's ascetic descent into identical lists of black in Material Design.
The one problem is the Alexa integration. You can add tasks to your shopping list or your to do list. The shopping list works great for me. I check it once a week when I do my grocery shopping and everyone knows to just tell Alexa when they need something. The to-do list is a disaster. These go to a separate to-do list project without a due date and I will never ever find them there. Anything else I add will end up in the Inbox with a due date of today so I'm forced to classify and if necessary reschedule it. Which is exactly what I want.
When I say 'Alexa, add x to my to-do list' I want that task to be visible. This integration design flaw could lead to at least one child growing up in an unfamiliar part of town without parents, or worse.
I emailed Todoist and they politely declined to change the way the integration works. After a brief period of steaming I've rolled up my sleeves and fixed it with their API. Which doesn't use OAuth so now I love them even more.
The script is below. Create a new Apps Script project in Google Drive (New and then choose More to find this) and copy in the code. You can get the API token from the bottom of the Integrations section in Todoist settings. Then just click the clock in the Apps Script project and schedule checkForAlexaTasksWithNoDate() to run as often as you need. The script will check the Alexa To-do List project and if anything is in there without a due date it will set it to today to force you to deal with it.
Updated 2022-12-06 21:27:
Todoist have updated their API to v2 (migration guide). There are no breaking changes for this code and so I got it working again by changing v1 to v2 in the API calls. The code above has also been changed to update dates in multiple projects, this is because I started checking for tasks with no due date in the Inbox project as well as Alexa tasks. You can delete the Inbox call if you don't want this behavior (and add additional projects if needed as well).
(Published to the Fediverse as:
Using the Todoist API to set a due date on the Alexa integration to-do list (with Apps Script) #code#appsscript#google#drive#todoist#productivity Script to fix an annoyance with the Todoist Alexa integration where tasks are added without a due date leading them to be overlooked. Requires Google Drive and Todoist.)
Google just released URL inspection as part of the Search Console API. I check for issues periodically in Search Console but it would be great to just get an email when an issue crops up. The Apps Script project below does just that by monitoring URLs from your sitemap for changes and sending an email whenever anything is detected. The Search Console API has a limit of 2,000 calls per day and Apps Script also imposes a time limit on scripts. The approach I take below assigns a random day of the week to each URL to limit the number checked on each run. Depending on the size of your site you may want to remove this check (or go the other way if you have a large number of URLs to monitor). Follow the steps below to get your own monitoring spreadsheet up and running.
Google Sheet
Create a new spreadsheet in Google Drive and call it anything you want. Rename an empty sheet to 'gsc', this sheet will store the index data. You don't need to make any other changes to this sheet.
Choose Apps Script from the Extensions menu. This will open up the script editor for your spreadsheet. I find that sometimes the editor opens with the wrong account if you're signed into more than one. If that applies to you, check quickly to make sure the right account is selected. With Code.gs selected copy and paste the script below replacing the default function:
There are a few configuration variables to enter at the top. AlertEmail is the email address to notify when index status changes are detected. SitemapUrl is the full URL of your sitemap. The current implementation does not support sitemap index files, this needs to be a regular sitemap containing URLs. SearchConsoleProperty should be the URL of the site to monitor, or sc-domain: followed by the domain for domain properties (for this site sc-domain:ithoughthecamewithyou.com).
Click Project Settings (the cog in the left hand menu of the script editor) and copy the Script ID. Make a note of this for later.
Create a new project (click the drop down to the right of 'Google Cloud Platform Console' and then New Project). Pick any name you like.
Once the project is created, find APIs and Services in the left hand menu and choose Library.
Search for Search Console, click on the Search Console API and then click Enable.
A new screen will load, click Credentials in the left hand menu.
Click Configure Consent Screen and choose the internal type.
Fill in the required fields - application name and contact emails.
Add the ./auth/webmasters.readonly scope for readonly access to Search Console data.
Once the consent screen is complete click Credentials in the left hand menu again.
Click Create Credentials at the top and choose OAuth Client ID.
Choose Web Application.
Add https://script.google.com/macros/d/{SCRIPTID}/usercallback to authorized redirect URLs, replacing {SCRIPTID} including the brackets with the Apps Script ID you noted above.
Make a note of the Project ID.
Complete the Apps Script
Return to the Apps Script project and find the settings page. Set the GCP project to the project ID you noted above. Also on this page check the Show "appsscript.json" manifest file in editor option.
Go to the code editor and open appsscript.json. Add the following line:
Make sure the script is saved and close the script editor window. Reload the spreadsheet. Once the reload completes you should have a Search Console menu at the top of the spreadsheet. Choose Update Data from the Search Console menu. This will run for a few minutes and then populate the 'gsc' sheet with your URL data. See UrlInspectionResult in the API documentation for more information about the meaning of each field. You should also get a lengthy email with a notification for each URL that was inspected. This will continue for the first week and then you'll only get updates for interesting status changes.
Scheduling
Now the project is working, open the script editor again (Apps Script from the Extensions menu) and open Triggers (the clock icon in the left hand menu). Click Add Trigger at the very bottom right of the window. Select runUpdate as the function to run. Change event source to time-driven, and then select day timer and an hour to run the script. Lastly click Save. Your Google Search Console monitor will now run every day, and if the index status of a page changes you'll get an email about it within a week. The spreadsheet will also come in handy for other analysis and reporting.
Troubleshooting
You might need to tweak the script if you start hitting limits. The URL inspection API currently has a 2,000 call / day quota. Apps Script will only run for around 7 mins on a free account and 20 mins if you have Google Workspace. If either of these limits apply you could modify the 'checkDay' logic to use day of the month (or year, or ...) to reduce the number of URLs inspected on each run. If you need to do this remember to update the Check Day column on the 'gsc' sheet as well.
The script assumes that the URLs you want to monitor are in your sitemap. If this is not the case you can add URLs to the sheet directly. As long as they are part of the configured property you will still get results. If you use this method you might want to comment out the updatePagesFromSitemap() call in runUpdate() to save time.
If anything else goes wrong please leave a comment below and I'll do my best to help you.
Updated 2022-02-08 17:40:
After a couple of days I have a full dump of my sitemap from the page index status API. I wrote this script for the alerting possibilities but couldn't resist some analytics once the dataset was complete.
The chart above shows sessions vs days since the last Google Crawl. Pretty stark - Google keeps a close eye on the pages it sends traffic to and not so much on the others.
I set lastmod honestly and there is good news here. I could only find two cases where Google had not crawled the page since the last modified date. So when the sitemap says a page has changed the odds are good that it will get another crack at the index. The two exceptions are unusual posts that are updated hourly and weekly respectively and both have been crawled recently.
The breakdown of index status matches Google Search Console pretty well but I have a handful of pages that are 'Indexed, not submitted in sitemap', even though they are in the sitemap and no such status is shown on Search Console. I don't know if this is a glitch in the index status API or something to do with how the pages were discovered. Some light searching suggests that this message is usually what you would expect it to be.
Lastly, updating the sheet for my site is more bound by script execution time than the API limits. I changed it to run every hour and instead of partitioning by day of week I used a random hour of the day which means I check every URL at least once every 24 hours.
Updated 2022-04-24 10:50:
I just updated the code and post above. I've had occasional issues where updating the sheet failed which caused the next run to go back to the beginning with no saved index status. To reduce the chance of this happening I've added some retry logic and also improved the speed of the sheet load and save functions. I also had a comment on the code that suggested an easier way to handle OAuth and have incorporated this in the new version.
(Published to the Fediverse as:
Monitor page index status with Google Sheets, Apps Script and the Google Search Console API #code#searchconsole#appsscript#google#sheets#drive#seo How to use Apps Script in Google Sheets to automatically monitor index status in Google Search Console and get an email alert if anything changes.)
