Summer Solstice 2022

Summer Solstice 2022

Summer starts right now (09:14 UTC June 21 2022) in the northern hemisphere, winter for those with latitude signum differences. Rendered in Catfood Earth.

(Related: Summer Solstice 2021; Summer Solstice 2017; Winter Solstice 2020)

(You might also like: Great Blue Heron; About Hikes; City by the Bay)

(All Code Posts)

Outlook/Office iCal feed 400 bad request error with C# WebClient

Updated on Tuesday, June 21, 2022

400

Just in case it helps someone else I was able to fetch an Outlook iCal feed using C#'s WebClient for years until it stopped working in June, 2022 with a 400 / bad request error. I was downloading a set of calendars and the fix was just to use a new WebClient for each calendar so it must be some kind of state thing in WebClient.

To regain around 1% of sanity I have a task that pulls the various calendars that life throws at me and combines them into a single, de-duplicated calendar. The Google Calendar on my phone is gorgeous as a result. My primary calendar is orange and then all of the miscellanea are teal and unique. It would be great if Google Calendar could do this without help, but it was worth the effort not to have some random soccer match repeated five times in different colors.

Last week one calendar, an Outlook feed, started failing with 400 bad request.

Naturally I assumed that the server had started to suddenly care about some header or other and I started playing around with setting User-Agent and various Accept headers without any luck. To make debugging slightly easier I moved the Outlook calendar out of a loop (where I was iterating through a list of iCal feeds that I need to be aware of) and then it magically started working. The magic in this case must be a fresh WebClient and so the fix was to use a new WebClient for each calendar instead of reusing a single instance. It looks like WebClient is deprecated in .NET 6 and one is supposed to start using HttpClient instead so that's probably another fix but not one I'm going to wrestle with today.

(Related: The Secret Diary of a Xamarin Android Developer, Aged 48 1/3; Export Google Fit Daily Steps, Weight and Distance to a Google Sheet; The curious case of the missing slugs (in BlogEngine.net 2.8))

(You might also like: Is Intuit Insane?; GetvCard shutting down at the end of 2013; Last Light)

(All Code Posts)

Catfood WebCamSaver 3.28

Updated on Friday, June 24, 2022

Long term solar powered time lapse camera using Arduino

Solar powered 5MP Arduino time lapse camera

After a few experiments I have a pretty decent platform for capturing long term time lapse footage. The system is designed to run for around a month and captures a photo every thirty seconds during daylight hours. It's a fairly cheap build so I'm not too worried about theft or accidental damage. This post contains build instructions and sample code to get the camera up and running. Before that, here's a quick video made from a few test runs:

On the hardware side you need an Arduino Mega 2560 REV3, the OV5642 camera module, an SD module and card, a battery and a solar panel. You'll also want a waterproof case, cables and some zip ties. Everything I used is on this Amazon list. I tried a few different power solutions and the Voltaic Systems 6W solar panel + 6,400 mAh battery combination is ideal. Their battery banks are 'always on' and continue to provide power even when the Arduino isn't doing anything. Most battery banks shut down when there is low power usage. In my testing the 6W panel managed to keep the system running with no issues.

The main limitation is that Arduino only supports up to 32GB SD cards. With the code below this allows around a month of photos. Depending on the application you could go much longer - the variables are interval (I'm shooting a frame around every 30 seconds) and file size (lower resolution and/or JPEG quality). At some point I'd like to figure out cellular upload but for now 32GB will have to do.

Assembly is pretty easy. The OV5642 camera uses both the I2C and SPI buses. For I2C connect the pins labeled SCL and SDA to SCL and SDA on the Arduino Mega (D21 and D20 pins - search Arduino Mega pinout if necessary). The SPI connections are MISO, MOSI and SCK (available at D50, D51 and D52). You also need CS (chip select) - wire this to D10. Finally wire ground and power to the GND and 5V pins. The SD module also uses SPI. Wire its MISO, MOSI and SCK to the same pins (D50, D51 and D52). SD needs a separate CS (chip select), use D7. And finally for SD connect ground and power to GND and 5V. That's it.

The box linked above is large enough to fit all the components easily. You'll need to drill two holes, one for the camera lens and one for the solar panel cable. Once assembled, seal around both as well as you can.

To program the Mega you need to install the Arduino IDE and then download and install the ArduCAM library. Make sure that the ArduCAM folder is directly under the libraries folder for your Arduino IDE (on Windows this is probably C:\Program Files (x86)\Arduino\libraries\ArduCAM). Then open the file memorysaver.h in the ArduCAM folder and make sure that only the OV5642 line is uncommented (#define OV5642_MINI_5MP_PLUS). If you get build errors it's almost certainly this step that is the problem so check carefully.

At this point I'd suggest trying the blink sample to make sure that you can connect to and program your board successfully. Once this is working create a new sketch and then copy the code below:

A few notes on the code. Near the top you can uncomment the line #define USE_SERIAL to get diagnostic messages written to the serial monitor. This can be useful for debugging if necessary. The sketch uses the built in LED to indicate problems as well - if everything is working you'll see the LED light up during setup and then switch off once the main loop starts. When a picture is written to the SD card the SD module LED will light up, so if you see that happening every 30 seconds or so then you should be in good shape. If the setup code fails to find the camera or SD module the internal LED will blink indicating that you need to fix something. Check that the OV5642 and SD modules are correctly wired. I would also try disconnecting power from the Mega for a few seconds. I have experienced connection issues with the OV5642 that are only fixed by a power cycle.

At startup there is also a check for existing files on the SD card and the file number is incremented by 1000 as needed until existing files will not be overwritten. This is useful in case the system loses power and restarts for any reason. I have included most configurable parameters for the camera module in the setup code like contrast and exposure adjustments so these should be easy to tweak based on your specific application.

Finally the startup code disables Mega features that are not used and after each photo the camera is powered down and the Mega put to sleep. Each sleep is around nine seconds. The number of sleeps depends on the size of the image - if below 200K it will sleep for around half an hour before trying again. This saves filling up the SD card with nighttime images and preserves power when the solar panel is unlikely to be helpful. This is another area to tweak depending on how you plan to use the system.

If you have any questions (or if you build this and it works for you) please leave a comment below.

(Related: Style Transfer for Time Lapse Photography; Capture DropCam (Nest Cam) frames to Google Drive; Export Google Fit Daily Steps, Weight and Distance to a Google Sheet)

(You might also like: Lizard; Skype for Android 4 - Better but Useless; San Francisco Street Tree Datalapse)

(All Code Posts)

Vernal (Spring) Equinox 2022

Spring Equinox 2022 rendered in Catfood Earth

Spring starts right now in the northern hemisphere, Autumn if you happen to find yourself south of the equator. Rendered in Catfood Earth.

(Related: Vernal (Spring) Equinox 2020; Vernal (Spring) Equinox 2019; Vernal Equinox 2015)

(You might also like: China Camp State Park; Memo to future: How to reassemble Kate’s cot; AMP!)

(All Code Posts)

Catfood WebCamSaver 3.27

Updated on Saturday, March 19, 2022

Catfood WebCamSaver 3.27

Catfood WebCamSaver 3.27 is available to download.

This release contains the latest webcam list and will upgrade any current set of webcams. I'm currently releasing updates for WebCamSaver every three months with the latest cams.

(Related: Catfood WebCamSaver; Catfood: WebCamSaver and PdfScan; Catfood Earth)

(You might also like: News: Google, Microsoft, Mozilla And Others Team Up To Launch WebAssembly, A New Binary Format For The Web; Año Nuevo; High-Frequency Trading)

(All Code Posts)

Using the Todoist API to set a due date on the Alexa integration to-do list (with Apps Script)

Todoist Alexa Fix

I love almost everything about Todoist. It's rich enough to scratch all my productivity itches while also being basic enough that I don't spend time gardening my tasks. Also, their Android app is just gorgeous, the exact opposite of Google's ascetic descent into identical lists of black in Material Design.

The one problem is the Alexa integration. You can add tasks to your shopping list or your to do list. The shopping list works great for me. I check it once a week when I do my grocery shopping and everyone knows to just tell Alexa when they need something. The to-do list is a disaster. These go to a separate to-do list project without a due date and I will never ever find them there. Anything else I add will end up in the Inbox with a due date of today so I'm forced to classify and if necessary reschedule it. Which is exactly what I want.

When I say 'Alexa, add x to my to-do list' I want that task to be visible. This integration design flaw could lead to at least one child growing up in an unfamiliar part of town without parents, or worse.

I emailed Todoist and they politely declined to change the way the integration works. After a brief period of steaming I've rolled up my sleeves and fixed it with their API. Which doesn't use OAuth so now I love them even more.

The script is below. Create a new Apps Script project in Google Drive (New and then choose More to find this) and copy in the code. You can get the API token from the bottom of the Integrations section in Todoist settings. Then just click the clock in the Apps Script project and schedule checkForAlexaTasksWithNoDate() to run as often as you need. The script will check the Alexa To-do List project and if anything is in there without a due date it will set it to today to force you to deal with it.

More Google Apps Script Projects

(All Code Posts)

Monitor page index status with Google Sheets, Apps Script and the Google Search Console API

Updated on Friday, May 20, 2022

GSC Monitor

Overview

Google just released URL inspection as part of the Search Console API. I check for issues periodically in Search Console but it would be great to just get an email when an issue crops up. The Apps Script project below does just that by monitoring URLs from your sitemap for changes and sending an email whenever anything is detected. The Search Console API has a limit of 2,000 calls per day and Apps Script also imposes a time limit on scripts. The approach I take below assigns a random day of the week to each URL to limit the number checked on each run. Depending on the size of your site you may want to remove this check (or go the other way if you have a large number of URLs to monitor). Follow the steps below to get your own monitoring spreadsheet up and running.

Google Sheet

Create a new spreadsheet in Google Drive and call it anything you want. Rename an empty sheet to 'gsc', this sheet will store the index data. You don't need to make any other changes to this sheet.

Choose Apps Script from the Extensions menu. This will open up the script editor for your spreadsheet. I find that sometimes the editor opens with the wrong account if you're signed into more than one. If that applies to you, check quickly to make sure the right account is selected. With Code.gs selected copy and paste the script below replacing the default function:

There are a few configuration variables to enter at the top. AlertEmail is the email address to notify when index status changes are detected. SitemapUrl is the full URL of your sitemap. The current implementation does not support sitemap index files, this needs to be a regular sitemap containing URLs. SearchConsoleProperty should be the URL of the site to monitor, or sc-domain: followed by the domain for domain properties (for this site sc-domain:ithoughthecamewithyou.com).

Click Project Settings (the cog in the left hand menu of the script editor) and copy the Script ID. Make a note of this for later.

API Console

Next we need to configure the Search Console API in the Google Cloud Platform Console.

  1. Create  a new project (click the drop down to the right of 'Google Cloud Platform Console' and then New Project). Pick any name you like. 
  2. Once the project is created, find APIs and Services in the left hand menu and choose Library.
  3. Search for Search Console, click on the Search Console API and then click Enable. 
  4. A new screen will load, click Credentials in the left hand menu. 
  5. Click Configure Consent Screen and choose the internal type.
  6. Fill in the required fields - application name and contact emails. 
  7. Add the ./auth/webmasters.readonly scope for readonly access to Search Console data.
  8. Once the consent screen is complete click Credentials in the left hand menu again.
  9. Click Create Credentials at the top and choose OAuth Client ID. 
  10. Choose Web Application.
  11. Add https://script.google.com/macros/d/{SCRIPTID}/usercallback to authorized redirect URLs, replacing {SCRIPTID} including the brackets with the Apps Script ID you noted above. 
  12. Make a note of the Project ID.

Complete the Apps Script

Return to the Apps Script project and find the settings page. Set the GCP project to the project ID you noted above. Also on this page check the Show "appsscript.json" manifest file in editor option.

Go to the code editor and open appsscript.json. Add the following line:

"oauthScopes": ["https://www.googleapis.com/auth/script.external_request", "https://www.googleapis.com/auth/webmasters.readonly", "https://www.googleapis.com/auth/spreadsheets.currentonly", "https://www.googleapis.com/auth/script.send_mail"],

Make sure the script is saved and close the script editor window. Reload the spreadsheet. Once the reload completes you should have a Search Console menu at the top of the spreadsheet. Choose Update Data from the Search Console menu. This will run for a few minutes and then populate the 'gsc' sheet with your URL data. See UrlInspectionResult in the API documentation for more information about the meaning of each field. You should also get a lengthy email with a notification for each URL that was inspected. This will continue for the first week and then you'll only get updates for interesting status changes.

Scheduling

Now the project is working, open the script editor again (Apps Script from the Extensions menu) and open Triggers (the clock icon in the left hand menu). Click Add Trigger at the very bottom right of the window. Select runUpdate as the function to run. Change event source to time-driven, and then select day timer and an hour to run the script. Lastly click Save. Your Google Search Console monitor will now run every day, and if the index status of a page changes you'll get an email about it within a week. The spreadsheet will also come in handy for other analysis and reporting.

Troubleshooting

You might need to tweak the script if you start hitting limits. The URL inspection API currently has a 2,000 call / day quota. Apps Script will only run for around 7 mins on a free account and 20 mins if you have Google Workspace. If either of these limits apply you could modify the 'checkDay' logic to use day of the month (or year, or ...) to reduce the number of URLs inspected on each run. If you need to do this remember to update the Check Day column on the 'gsc' sheet as well.

The script assumes that the URLs you want to monitor are in your sitemap. If this is not the case you can add URLs to the sheet directly. As long as they are part of the configured property you will still get results. If you use this method you might want to comment out the updatePagesFromSitemap() call in runUpdate() to save time.

If anything else goes wrong please leave a comment below and I'll do my best to help you.

Updated 2022-02-08 17:40:

Monitor page index status with Google Sheets, Apps Script and the Google Search Console API

After a couple of days I have a full dump of my sitemap from the page index status API. I wrote this script for the alerting possibilities but couldn't resist some analytics once the dataset was complete.

The chart above shows sessions vs days since the last Google Crawl. Pretty stark - Google keeps a close eye on the pages it sends traffic to and not so much on the others.

I set lastmod honestly and there is good news here. I could only find two cases where Google had not crawled the page since the last modified date. So when the sitemap says a page has changed the odds are good that it will get another crack at the index. The two exceptions are unusual posts that are updated hourly and weekly respectively and both have been crawled recently.

The breakdown of index status matches Google Search Console pretty well but I have a handful of pages that are 'Indexed, not submitted in sitemap', even though they are in the sitemap and no such status is shown on Search Console. I don't know if this is a glitch in the index status API or something to do with how the pages were discovered. Some light searching suggests that this message is usually what you would expect it to be.

Lastly, updating the sheet for my site is more bound by script execution time than the API limits. I changed it to run every hour and instead of partitioning by day of week I used a random hour of the day which means I check every URL at least once every 24 hours.

Updated 2022-04-24 10:50:

I just updated the code and post above. I've had occasional issues where updating the sheet failed which caused the next run to go back to the beginning with no saved index status. To reduce the chance of this happening I've added some retry logic and also improved the speed of the sheet load and save functions. I also had a comment on the code that suggested an easier way to handle OAuth and have incorporated this in the new version.

More Google Apps Script Projects

(All Code Posts)

Catfood Earth for Android 4.20

Updated on Tuesday, January 25, 2022

Catfood Earth for Android 4.20

Catfood Earth for Android 4.20 adds support for Material You, the custom color scheme introduced in Android 12. By default every wallpaper update will also subtly change your system color palette. You can override this from settings -> Wallpaper & style and switch back to basic colors if you don't like the Catfood Earth colors. This update was surprisingly painful. You can install Catfood Earth from Google Play, or for existing users it will automatically update in the next day or so.

(Related: Catfood Earth; Improving the accuracy of the new Catfood Earth clouds layer; Catfood WebCamSaver)

(You might also like: Moth; Coronavirus Hikes: June 2020; Baby tech should let everyone sleep)

(All Code Posts)

The Secret Diary of a Xamarin Android Developer, Aged 48 1/3

WallpaperColors

I have been trying to update Catfood Earth for Android to support Material You in Android 12. This sets a color palette from your wallpaper and other than making notifications harder to manage seems to be the main thing that Google has been working on for the past year. Live wallpaper isn't automatically supported, it's up to you to tell the system about your colors.

Happily there is an overridable OnComputeColors in WallpaperService.Engine, and you can create a WallpaperColors object from a bitmap so this looked like a five minute update. Hahahahahahhahahahah.

I spent a few weeks waiting for the Android 12 SDK to be available in Xamarin. I found some pointers to their GitHub and assumed it would show up there and so waited a few weeks. It still isn't there. I found some article about forcing it to install in Visual Studio 2019 that didn't work for me, and then realized that Visual Studio 2019 update checks were crashing. While trying to fix that I found that Visual Studio 2022 has been released and this installs with the Android 12 SDK!

These days I work on side projects when I get the occasional free hour. Almost inevitably that hour is consumed with updating two or three things and then hoping that the next time I get some time at least I'll be ready to go. But when that hour arrives I'm back to updating again.

So finally I have the right SDK and drop in an OnComputeColors and a call to NotifyColorsChanged when my wallpaper is updated. Time to start testing. The good news is that Material You is now working with Catfood Earth selected as the current wallpaper. The bad news is that every time I call NotifyColorsChanged the launcher disappears and I'm just left with the wallpaper and no icons or search box. Probably not a good experience. I try moving the NotifyColorsChanged call around to different points in code and it makes no difference. This simple update is rapidly spiraling from minutes to days to weeks.

Maybe it's some Xamarin bug, or something in Android 12 or maybe a change in the solemn contract between live wallpaper services and the rest of the system. Can't find any hint of any of this on the Internet.

It's also possible that the bitmap I'm passing is too large or too complex or too something else. So let's try just sending back a simple WallpaperColors in OnComputeColors and see if that helps.

The Android documentation for WallpaperColors suggests that you can construct it from three colors which seems sensible. The Xamarin implementation takes ColorObject instead of Color. What is a ColorObject? The sparse Xamarin documentation suggests that it has an empty constructor that creates an opaque black color and then properties like Red that are read only. So an immutable black? That can't be right? When constructing a color the constructor would seem like a great place to tell the class what color it is. Failing that, a FromColor or FromArgb would seem to follow the right sort of convention. But no. You need to find ValueOf. To create a color that is a class and not a value. Sigh. Maybe that is some kind of deep Android convention that I would just know if I did this more often than once a year or so.

Although, maybe most Google Developers can't figure out how to make a non-black color and that's the origin of Material Design 2. Everything else is just marketing rationalization on top of a terrible API.

Finally I have a simple WallpaperColors to test with but my build has broken. At some point while figuring out how to get a ColorObject that isn't black I thought a NuGet package would help. My project is now completely broken. My experience with touching NuGet is that you can either spend several days unpicking the carnage or just start over. Git reset --hard  HEAD it is then.

After quickly reimplementing the changes so far, returning a vanilla WallpaperColors object works... and so does the FromBitmap version. So 90% chance that I bodged something subtle the first time and didn't make the mistake on the second pass. 10% chance it was something strange in Visual Studio.

(Related: Got It; Export Google Fit Daily Steps, Weight and Distance to a Google Sheet; Outlook/Office iCal feed 400 bad request error with C# WebClient)

(You might also like: Summer Solstice 2020; Lottocracy vs Legislative Service; Build Back Betterer)

(All Code Posts)

I Thought He Came With You is Robert Ellison's blog.

Newsletter