(Published to the Fediverse as:
Autumnal Equinox 2018 #code#earth#equinox#autumnal The moment of Autumnal Equinox 2018 (the start of Autumn in the Northern Hemisphere) rendered in Catfood Earth.)
By Robert Ellison. Updated on Tuesday, February 13, 2024.
I have been experimenting with word2vec recently. Word2vec trains a neural network to guess which word is likely to appear given the context of the surrounding words. The result is a vector representation of each word in the trained vocabulary with some amazing properties (the canonical example is king - man + woman = queen). You can also find similar words by looking at cosine distance - words that are close in meaning have vectors that are close in orientation.
This sounds like it should work well for finding related posts. Spoiler alert: it does!
My old system listed posts with similar tags. This worked reasonably well, but it depended on me remembering to add enough tags to each post and a lot of the time it really just listed a few recent posts that were loosely related. The new system (live now) does a much better job which should be helpful to visitors and is likely to help with SEO as well.
I don't have a full implementation to share as it's reasonably tightly coupled to my custom CMS but here is a code snippet which should be enough to get this up and running anywhere:
The first step is getting a vector representation of a post. Word2vec just gives you a vector for a word (or short phrase depending on how the model is trained). A related technology, doc2vec, adds the document to the vector. This could be useful but isn't really what I needed here (i.e. I could solve my forgetfulness around adding tags by training a model to suggest them for me - might be a good project for another day). I ended up using a pre-trained model and then averaging together the vectors for each word. This paper (PDF) suggests that this isn't too crazy.
For the model I used word2vec-slim which condenses the Google News model down from 3 million words to 300k. This is because my blog runs on a very modest EC2 instance and a multi-gigabyte model might kill it. I load the model into Word2vec.Tools (available via NuGet) and then just get the word vectors (GetRepresentationFor(...).NumericVector) and average them together.
I haven't included code to build the word list but I just took every word from the post, title, meta description and tag list, removed stop words (the, and, etc) and converted to lower case.
Now that each post has a vector representation it's easy to compute the most related posts. For a given post compute the cosine distance between the post vector and every other post. Sort the list in ascending order and pick however many you want from the top (the distance between the post and itself would be 1, a totally unrelated post would be 0). The last line in the code sample shows this comparison for one post pair using Accord.Math, also on Nuget.
I'm really happy with the results. This was a fast implementation and a huge improvement over tag based related posts.
Updated 2023-01-29 22:00:
I have recently moved this functionality to use an OpenAI embeddings API integration.
(Published to the Fediverse as:
Better related posts with word2vec (C#) #code#software#word2vec#ithcwy#ml How to use word2vec to create a vector representation of a blog post and then use the cosine distance between posts to select improved related posts.)
By Robert Ellison. Updated on Saturday, February 19, 2022.
Update April 19, 2020: I made a longer, higher resolution version which you can find here.
Here's an animation showing a year of global cloud cover (from July 2017 to July 2018) :
The clouds are sourced from the free daily download at xplanet. I run a Google apps script that saves a copy of the image to Google Drive every day (basically the same as this script to save Nest cam images). The hard part was waiting a year to get enough frames. Xplanet combines GEOS, METEOSAT and GMS satellite imagery with some reflection near the poles. Although I didn't need to for this project note that you can subscribe to higher quality / more frequent downloads.
As well as the clouds you can also see the terminator between day and night change shape over the course of the year. This video starts and ends with the Summer equinox when days are longest in the Northern hemisphere.
Where it's nighttime the image is based on NASA's Black Marble. The daytime is based on Blue Marble, but blended with a different older image which has better ocean colors and interpolated daily between twelve monthly Blue Marble satellite images. The result of this is that you can see snow and ice coverage changing over the course of the year. If you look closely you'll also notice vegetation growing and dying back with the seasons.
Rendered in a slightly modified build of Catfood Earth (the main release doesn't know how to access my private cache of xplanet cloud images). As well as combining day, night and cloud images Catfood Earth can also show you earthquakes, volcanoes, US weather radar, political borders, places and time zones. It has been enlivening Windows desktop wallpaper for fifteen years now (as shareware back when that was a thing, these days it's a free download for Windows and Android).
Summer starts now in the Northern Hemisphere and the Sun is at its highest point in the sky. For those in the Southern Hemisphere I'm sorry to report that the opposite is true. Rendered in Catfood Earth.
(Published to the Fediverse as:
Summer Solstice 2018 #code#solstice#summer#winter#earth#northern#estival Summer (or estival) Solstice 2018 rendered in Catfood Earth. This means the start of Summer in the Northern Hemisphere and Winter for the Southern.)
Catfood Earth 3.43 updates the timezone database to 2018e. The big change is that North Korea is moving back to UTC +09 today (May 5, 2018). The time zones layer in Catfood Earth shows the current time in each zone at the top of your screen and color codes each country and region. You can also display a list of specific places (either an included list of major cities or your own custom locations).
Catfood Earth is dynamic desktop wallpaper for Windows that includes day and night time satellite imagery, the terminator between daytime and nighttime, global cloud cover, time zones, political borders, places, earthquakes, volcanoes and weather radar. You can choose which layers to display an how often the wallpaper updates.
(Published to the Fediverse as:
Vernal (Spring) Equinox 2018 #code#earth#equinox#spring#autumn#vernal Catfood Earth render of the moment that Spring starts in 2018 (the Vernal Equinox), or Autumn if you live south of the Equator.)
By Robert Ellison. Updated on Saturday, November 9, 2024.
Google Fit is a great way to keep track of your daily step count without needing to carry a Fitbit or other dedicated tracker. It's not easy to get that data out though, as far as I can tell the only way is Google Takeout which is not made for automation. Luckily there is an API and you can do almost anything with Google Sheets.
If you're looking to export your step count, weight and distance this post has everything you need, just follow the instructions below to get your spreadsheet up and running. This is also a good primer on using OAuth2 with Google Apps Script and should be a decent starting point for a more complex Google Fit integration. If you have any questions or feedback please leave a comment below.
To get started you need a Google Sheet, an apps script project attached to the sheet and a Google API Project that will provide access to the Fitness API. That might sound intimidating but it should only take a few minutes to get everything up and running.
In Google Drive create a new spreadsheet and call it whatever you like. Rename the first tab to 'Metrics'. Enter 'Date' in cell A1, 'Steps' in B1, 'Weight' in C1 and 'Distance' in D1. To grab history as well create another tab called 'History' with the same headers. Next select 'Script editor...' from the Tools menu which will open a new apps script project.
Give the apps script project a name and then select 'Libraries...' from the Resources menu. Next to 'Add a library' enter 1B7FSrk5Zi6L1rSxxTDgDEUsPzlukDsi4KGuTMorsTQHhGBzBkMun4iDF and click Add. This will find the Google OAuth2 library. Choose the most recent version (24 at the time of writing) and click Save. Then select 'Project properties' from the File menu and make a note of the Script ID (a long series of letters and numbers).
Open the Google API Console. Create a new project and name it something like 'Google Fit Sheet'. From the Dashboard click Enabled APIs and services and find and select the Fitness API. Then go to Credentials and create an OAuth Client ID. You'll be asked to create a consent screen, the only field you need to enter is the product name (i.e. 'My Fit App'). Then choose Web Application as the application type. You need to set the name and the authorized redirect URL. The redirect URL is https://script.google.com/macros/d/{SCRIPTID}/usercallback replacing {SCRIPTID} with the actual Script ID you made a note of above. After adding this make a note of the Client ID and Client Secret.
Go back to the apps script project and paste the code below into the Code.gs window:
Right at the top of the code there are spaces to enter the Client ID and Client Secret from the API Console. Enter these and save the project.
Switch back to your Google Sheet and reload. After reloading there will be a Google Fit menu item. First select Authorize... You'll get a screen to authorize the script and then a sidebar with a link. Click the link to authorize the script to access your Google Fit data. You can then close the sidebar and select Get Metrics for Yesterday from the Google Fit menu. You should see a new row added to the spreadsheet with yesterday's date and fitness data.
The final step is to automate pulling in the data. Go back to the apps script project and select Current project's triggers from the Edit menu. Add a trigger to run getMetrics() as a time driven day timer - I recommend between 5 and 6am. You can also click notifications to add an email alert if anything goes wrong, like your Google Fit authorization expiring (in which case you just need to come back and authorize from the Google Fit menu again.
At this point you're all set. Every day the spreadsheet will automatically update with your step count from the day before. You can add charts, moving averages, export to other systems, pull in your weight or BMI, etc. I want to add a seven day moving average step count to this blog somewhere as a semi-public motivational tool... watch this space.
Note that weight will be blank in the spreadsheet for days with no weight data. Google Fit doesn't return the last known weight, only the known value for days where an update was recorded.
If you are looking to extend this sample to other data types then this API explorer page is very helpful for finding data types that the API documentation doesn't list.
A couple of times working on this script I got my authorization in a bad state and started getting a 400 error response from the API. If this happens run your Google Fit app, click the Profile icon at the bottom and then the Settings icon at the top right. Click Manage connected apps and then disconnect the script from Google Fit. Finally run the Reset Settings option from the menu in the sheet and then authorize again.
I updated this post on Jan 21, 2019 to extend the sample to handle weight and distance as well as steps. I also improved the history function to handle many days in one API call rather than a quick hack I added earlier that pulled a day at a time. I'd recommend using the code above rather than anything included in comments below (at least comments before this update).
(Published to the Fediverse as:
Export Google Fit Daily Steps, Weight and Distance to a Google Sheet #code#software#fit#appsscript#google#sheets#drive Detailed instructions for setting up an automatic daily export of your step count from Google Fit to a Google Sheet. Uses Google Sheets, Apps Script and the Google Fitness API.)
By Robert Ellison. Updated on Saturday, August 21, 2021.
Autumn/Fall starts now in the Northern Hemisphere. Springtime for the hemispherically challenged. Rendered in Catfood Earth (Windows, Android live wallpaper).
(Published to the Fediverse as:
Autumnal Equinox 2017 #code#earth#equinox#autumnal Autumnal (fall) equinox 2017. Autumn begins in the Northern Hemisphere, Spring in the Southern Hemisphere. Image rendered in Catfood Earth.)
